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Overview 


Family Rewards was an innovative approach to poverty reduction in the United States that was 
modeled on the conditional cash transfer (CCT) programs common in lower- and middle-income 
countries. The program offered cash assistance to poor families, contingent on their meeting certain 
criteria related to family health care, children’s education, and parents’ work, in the hope of reducing 
long-term poverty. The first version of Family Rewards was evaluated in New York City starting in 
2007 and had positive effects on some outcomes. The lessons learned from that study led to the next 
iteration and test of the model (“Family Rewards 2.0”), the subject of this report. 

Family Rewards 2.0 was launched in July 2011 in the Bronx, New York, and Memphis, Tennessee. 
While still offering rewards in the areas of children’s education, family health, and parents’ work, 
Family Rewards 2.0 offered fewer rewards in each domain, offered the education rewards only to 
high school students, made the rewards more timely by paying them each month, and included 
personalized family guidance. The addition of guidance from staff members, who actively helped 
families develop strategies to earn rewards, represented the biggest change to the original model. 

MDRC evaluated the program through a randomized controlled trial involving about 1 ,200 families 
in each city, half of whom could participate in the program and half of whom could not. This report 
presents findings on the program’s effects through four years. 

Key Findings 

Family Rewards transferred over $6,200, on average, to each participating family during the period 
in which it operated, or just over $2,000 per program year. Through Year 4, it had produced some 
positive effects on some outcomes, but left many other outcomes unchanged. 

• After some start-up challenges, the program was generally implemented well in both cities by 
Year 2. The guidance component became more intensive over time, although the amount of in- 
teraction between staff and participants was still less than envisioned, especially in Memphis. 

• Family Rewards 2.0 increased income and reduced poverty during the program period, and led 
to improvements in parents’ reports of life satisfaction and happiness. 

• The program’s primary effect in the health care area was an increase in preventive dental visits, 
although there is some evidence that it also improved adults’ self-reported health status, particu- 
larly for those in poorer health at study entry. 

• The program led to a reduction in employment covered by the unemployment insurance system, 
driven largely by reductions in work in Memphis. 

• The program did not affect students’ school progress through Year 4, either for the full sample 
of students or for the subgroup of academically proficient students. 

The findings show that Family Rewards 2.0, while replicating many of the results from the first 
model, did not prove to be more effective. Together, however, both studies provide important 
lessons about the operation and effects of a comprehensive CCT strategy in the United States. 
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Preface 


Conditional cash transfer (CCT) programs are an innovative approach to poverty reduction. 
They offer cash assistance to poor families, reducing current poverty, but condition this assis- 
tance on families’ efforts to get preventive health care and further their children’s education, 
with the hope of reducing longer-term poverty. They are fairly common in low- and middle- 
income countries but had never been tested in the United States until several years ago, when 
New York City launched Opportunity NYC-Family Rewards (referred to in this report as 
“Family Rewards 1.0”), a three-year demonstration sponsored by the New York City Center for 
Economic Opportunity (CEO). MDRC helped design the initiative based on several existing 
international programs and conducted the implementation and impact evaluations. Low-income 
families in several high-poverty neighborhoods in New York City were offered cash incentives 
(“rewards”) for completing activities related to children’s education, family preventive health 
care, and adults’ work or training. The program had few effects on most behavioral outcomes 
overall, but did reduce hardship, increase receipt of dental care, and improve the educational 
outcomes of more academically prepared high school students. Early lessons from the imple- 
mentation of that first demonstration, with its large number of incentives, together with patterns 
in its early impact findings suggested to CEO and MDRC that the model should be revised and 
tried again, this time with the support of the federal Social Innovation Fund. 

The new version of Family Rewards (“Family Rewards 2.0”) reduced the number of 
rewards the program offered, targeted the education rewards only to high school students, and 
paid the rewards more frequently. Most important, it provided more proactive guidance to help 
families earn rewards, something that the first demonstration suggested many families needed. 
Family Rewards 2.0 was tested again in New York City (the Bronx) as well as in Memphis, 
Tennessee, providing an opportunity to study the concept in a local context very different from 
New York City. 

The findings show that Family Rewards 2.0 replicated some of the effects found in 
Family Rewards 1.0, but fell short in other ways. It increased income and reduced poverty, for 
example, but had few effects on students’ progress in school, even for more academically 
prepared students. Although these results are disappointing, the findings do raise important 
policy questions. Both iterations of Family Rewards did succeed in reducing poverty and 
economic hardship, in part because many families received rewards without changing their 
behavior — most families already used preventive medical care, for example, and many 
students either had high attendance or earned rewards for other education milestones. But the 
program did not increase those rates substantially. In some cases, the attendance and health 
insurance rates among control group families were also high, often leaving less room to make a 
difference. In other cases, the incentives were not enough to change the trajectories of students 
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with low attendance and performance. Further complicating interpretation, these tests of 
conditional cash transfers operated on top of the existing safety net, not as an alternative to it, as 
is the case for CCT programs in Mexico and other Latin American countries. Last, in the United 
States, income transfer programs are conditioned on work. When unemployment rates soared 
during the Great Recession, demonstrating the limits of an existing policy that conditions 
benefits on work, a strategy that conditioned benefit receipt on a broader array of behaviors had 
appeal. 


Although the findings now available from both versions of Family Rewards provide 
important lessons about the use of incentives and CCTs in the United States to combat both 
short- and long-term intergenerational poverty, questions remain about the possible effects of a 
conditional cash transfer program if it were not structured as an add-on to the existing American 
system — specifically, whether it would lead to larger behavioral changes while still providing 
an adequate safety net for the most vulnerable families. 

Gordon L. Berlin 
President, MDRC 
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Executive Summary 


Family Rewards was an innovative approach to poverty reduction in the United States that was 
modeled on the conditional cash transfer (CCT) programs common in lower- and middle- 
income countries. The program offered cash assistance to low-income families, provided that 
they met certain conditions related to family health care, children’s education, and parents’ 
work. 


The first version of Family Rewards, called Opportunity NYC-Family Rewards (“Fam- 
ily Rewards 1.0”), was evaluated in New York City beginning in 2007 using a randomized 
controlled trial, in which families were randomly assigned to a program group that was offered 
the program or a control group that was not. While the program did not lead to effects on most 
major outcomes, it did lead to sizable reductions in poverty while the cash assistance was being 
offered and increased graduation rates for a group of ninth-graders who were more academical- 
ly prepared at study entry. 1 Early lessons and insights from that evaluation suggested that such 
an intervention might be made more effective if the model were modified in certain ways. 2 The 
lessons learned from that evaluation led to the next iteration and test of the model — “Family 
Rewards 2.0.” 

Family Rewards 2.0, the subject of this report, was launched in July 2011 in the Bronx, 
New York, and Memphis, Tennessee. While still offering rewards in the areas of children’s 
education, family health, and parents’ work, Family Rewards 2.0 refined the original model in 
several ways: it offered fewer rewards in each domain, paid those rewards more frequently, 
offered the education rewards only to high school students, and offered proactive and personal- 
ized guidance to help families earn rewards. The addition of guidance from staff members, who 
actively helped families develop strategies to earn rewards, represented the biggest change from 
the original model. 

This report examines whether those changes led to bigger impacts and whether the pro- 
gram had similar effects in a context different from New York City. The findings show that the 
new program achieved many of the same effects as the original model, but fell short in other, 
important ways. Family Rewards 2.0 met its short-term goals of increasing income and reduc- 
ing poverty, although the effects were smaller, given that less money was transferred overall. 


'James Riccio, Nadine Dechausay, David Greenberg, Cynthia Miller, Zawadi Rucks, and Nandita Verma, 
Toward Reduced Poverty Across Generations: Early Findings from New York City ’s Conditional Cash 
Transfer Program (New York: MDRC, 2010); James Riccio, Nadine Dechausay, Cynthia Miller, Stephen 
Nunez, Nandita Verma, and Edith Yang, Conditional Cash Transfers in New York City: The Continuing Story 
of the Opportunity NYC-Family Rewards Demonstration (New York: MDRC, 2013). 

2 Riccio et al. (2010); Riccio et al. (2013). 


ES-1 



(The shorter list of rewards meant that families had fewer ways of earning cash transfers.) The 
program also increased dental visits and adults’ self-reported health status, particularly for those 
in poorer health at study entry. Similar to the earlier program, the new model led to reductions 
in work and earnings for some participants. However, the new program did not affect students’ 
school progress through Year 4, neither for the full sample of students nor for a more academi- 
cally prepared subgroup. Overall, the findings indicate that Family Rewards 2.0 did not lead to 
bigger or more widespread effects. In addition, the failure to replicate the positive effects on 
school progress for more academically prepared students suggests that the model’s effects on 
education were not very robust. 

The Social Innovation Fund of the Corporation for National and Community Service 
and private foundations funded the Family Rewards 2.0 demonstration, and The Mayor’s Fund 
to Advance New York City, the New York City Center for Economic Opportunity, and MDRC 
managed it. The Children’s Aid Society operated the program in partnership with two commu- 
nity-based organizations in each city (the local chapter of Children’s Aid Society and Bronx- 
Works in the Bronx, and Memphis HOPE and Porter-Leath in Memphis). MDRC conducted the 
evaluation. 

The Program Model: 1.0 to 2.0 

Similar to its predecessor, Family Rewards 2.0 is based on the assumption that for a variety of 
reasons families may underinvest in their own development even though such investments can 
have long-term benefits. Financial incentives can help change their calculations, encouraging 
them to make extra investments of time and energy in certain educational, health care, and 
work-related efforts. Furthermore, the extra resources can help make it more feasible for low- 
income people to undertake such efforts in the short term, functioning not only as incentives for 
action but also as enabling resources. 

The original Family Rewards model offered families 22 cash rewards, covering activi- 
ties and outcomes in three domains. In the education domain, rewards were offered for parents’ 
attendance at parent-teacher conferences and for children’s attendance in school, satisfactory 
performance or better on standardized tests, completion of adequate credits per year, passing of 
exams required to graduate, and graduation. In the health domain, rewards were offered for 
maintaining health coverage and for preventive medical and dental checkups for each family 
member. In the work domain, parents were offered rewards for sustaining full-time work and 
for pursuing education or training while working. Families received payments for rewards 
earned every two months. Families were not offered services or counseling, since the goal was 
to test a pure incentives model. 

During the three years that Family Rewards 1.0 operated, the average participating fam- 
ily earned nearly $9,000 in rewards, or roughly $3,000 in each year, leading to large reductions 
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in poverty and reductions in a number of material hardships. The program did not affect school 
outcomes for elementary or middle school students, but did improve outcomes for the ninth- 
graders in the study who were performing at a proficient level or better academically when they 
entered, as measured on their prior year’s standardized tests, with sizable effects (for initially 
reading-proficient ninth-graders) on grade promotion and on graduation. In the health area, 
early, positive effects on visits to the doctor and health status faded, although there were 
continued impacts on health coverage and, especially, dental visits. Finally, the program led to a 
small increase in survey-reported employment, but to reductions for some participants in 
employment reported by the unemployment insurance (UI) system. 

Family Rewards 2.0 included rewards for the following milestones (Table ES.l): 

• Education: Students were rewarded for high attendance, good grades, satis- 
factory performance or better on state core exams, and taking college en- 
trance exams. 

• Flealth: Families received payments for obtaining medical and dental check- 
ups for each family member. 

• Work and training: Parents received payments for full-time work and for 
earning General Educational Development (GED) certificates. 

The program made several important modifications to the original Family Rewards 
model. In an effort to make the program easier to understand and focus families’ attention on 
a limited number of outcomes, the program offered 8 rewards across the three domains, 
instead of 22 rewards. While all children in a family were eligible to earn health rewards, the 
education rewards were only offered to high school students, given that the original program 
had no effects for younger students. The new program made the rewards more timely, and 
thus more salient to families, in two ways: first, by paying families monthly for rewards 
earned, rather than every two months; and second, by rewarding students for passing grades. 
The rewards for grades were also structured in an attempt to engage less academically 
proficient students, with rewards offered for A, B, and C grades. All education rewards 
earned were also paid directly to the students’ accounts, whereas in the first model parents 
received some portion of education rewards. 

Most notably, however, the new model offered proactive and personalized guidance to 
help families earn rewards. In the original program, staff members were purposely restricted 
from engaging in “case management,” so that the experiment would test the effects of the 
incentives offer alone. They were permitted to provide advice and referrals to participants when 
asked for assistance with services, but they did not carry individual caseloads, nor were they 
allowed to make proactive efforts to engage individual families in conversations about strategies 
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Table ES.l 


Family Rewards 2.0 Schedule of Rewards 


Domain 

Bronx 

Memphis 

Education incentives 
(high school students only) 



Attends 95% of scheduled 
school days 

$40 per month 

$40 per month 

Takes an SAT or ACT exam 
(once during program) 

$50 

$50; must score 19 or more 
on the ACT if administered 
by Memphis City Schools 3 

Receives grades on an official 
report card 1 ’ 

$30 per A (90-100); $20 per 

B (80-89); $10 per C (75-79) 

$30 per A (93-100); $20 per 

B (85-92); $10 per C (75-84) 

Passes up to 5 Regents exams c 
or 7 End-of-Course exams d 

$500 per Regents exam for a 
score of 75 or above; $400 
per exam for a score of 65-74 

$200 per End-of-Course 
exam for a score of proficient 
or advanced (increased to 
$300 in Year 3) 

Health incentives (parents and 
children 19 years and younger) 



Annual medical checkup 

$100 per family member 

$100 per family member 

Preventive dental care every 

6 months (once per year for 
children 1-5 years) 

$100 per family member 

$100 per family member 

Work incentives (parents only) 



Sustains frill-time employment 

$150 per month 

$150 per month 

Earns a GED certificate 

$400 

$400 


SOURCE: Children’s Aid Society’s Family Rewards program materials. 


NOTES: GED = General Educational Development. 

The ACT is scored out of 36. Memphis City Schools officials requested that the minimum score for the 
reward be set at 19 for students taking the test for free in class because this score is considered an indication 
that students are ready for college-level work. 

b In Year 1, the amount of each reward for grades was prorated based on the number of official report cards 
issued by a student's school. To simplify verification in Year 2, students were paid the listed amounts for their 
grades regardless of the number of official report cards they received, up to a maximum of $600 per program 
year. 

(continued) 
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Table ES.l (continued) 


c High school students (grades 9-12) in the Bronx were eligible to earn rewards for the following Regents 
exams: English, one of any math exams (including Math A, Math B, Integrated Algebra, Geometry, and 
Algebra 2/Trigonometry), U.S. History and Government, Global History and Geography, and one of any 
science exams (including Living Environment, Chemistry, Physics, and Earth Science). 

d High school students (grades 9-12) in Memphis were eligible to earn rewards for the following End-of- 
Course exams: Algebra 1, Algebra 2, Biology, English 1, English 2, English 3, and U.S. History. 


to earn rewards. Findings from the first evaluation suggested that many families needed — and 
wanted — more help to reach the relevant milestones. Consequently, under the new model, staff 
members at the community-based partner organizations were instructed to develop a “Family 
Earning Plan” with every family and meet with them at least twice per year to discuss their 
progress toward earning the rewards. Staff members were also directed to make more aggres- 
sive and frequent efforts to engage families who, according to program tracking data, were not 
earning many rewards. Each staff member carried a caseload of about 100 families. 

Children’s Aid Society managed the operations of Family Rewards 2.0 and provided 
technical assistance and oversight to the four Neighborhood Partner Organizations (NPOs) 
selected to implement the program. The NPOs were charged with implementing core compo- 
nents of the program. They recruited and enrolled families into the research sample, oriented 
families to the program, and provided continuing guidance to help families earn rewards. 

The Study Sample 

Family Rewards 2.0 was evaluated using a randomized controlled trial. In each city, about 
1,200 families were recruited for the study. Half were randomly assigned to a program group, 
offered Family Rewards, and half were assigned to a control group, not offered the program. 
The program targeted families with at least one child entering ninth grade or tenth grade. Once 
enrolled in Family Rewards 2.0, however, all of the family’s school-age children became 
eligible for the health-related rewards. The program also targeted recipients of benefits from 
Temporary Assistance for Needy Families (TANF) and the Supplemental Nutrition Assistance 
Program (SNAP, or “food stamps”), in order to get resources to the neediest families and to 
explore how a CCT approach might interact with and complement these safety net programs. 

Recruitment began in August 2011 in the Bronx and in September 201 1 in Memphis, 
when the NPOs received lists of potentially eligible participants provided by the human services 
agencies and departments of education in each city. The pace of recruitment varied between the 
cities due to delays in receiving some lists and difficulties in reaching some potential partici- 
pants. Enrollment was originally expected to conclude by October 2011, but in practice the 
majority of families in the study enrolled after that point. Rewards were intended to be offered 
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for three years, starting in September 2011. However, because many families entered the study 
after September 2011, the program period was extended beyond three years, to December 2014, 
in order to provide these late-entering families with at least three years of rewards. The evalua- 
tion tracked families for up to four years to determine the effects of the program on poverty, 
children’s education, family health, and parents’ work. 

The majority of families who enrolled in the study in both cities were single-parent 
families, with a higher percentage in Memphis (91 percent) than in the Bronx (77 percent). A 
large fraction of the sample in the Bronx was Hispanic (74 percent), while nearly all participat- 
ing families in Memphis were African American (98 percent). Adults in the Bronx had some- 
what lower education levels when they enrolled than adults in Memphis. For example, nearly 
half of the adults in the Bronx did not have a high school diploma or equivalency certificate 
when they enrolled in the study, compared with only 31 percent of adults in Memphis. In 
contrast, adults in the Bronx were more likely to be working when they entered the study (57 
percent) than their counterparts in Memphis (44 percent). 

Findings 

• After some start-up challenges, the program was generally implemented 
well in both cities by Year 2. The guidance component became more in- 
tensive over time, although the amount of interaction between staff and 
participants was still less than envisioned, especially in Memphis. 

The biggest innovation in the design of Family Rewards 2.0 was the addition of the 
family guidance component, and that component evolved considerably during the first two 
years of the program. Designated staff, or advisors, focused mainly on building relationships in 
the first year. One-on-one sessions at this time were largely transactional or customer service- 
oriented. Recognizing that advisors needed more training to go beyond customer service and 
encourage behavioral change, the NPOs introduced a form of counseling called motivational 
interviewing at the start of Year 2. Motivational interviewing is a widely used, client-centered 
directive approach to counseling in which a counselor uses a set of techniques to help partici- 
pants identify their goals, explore any ambivalence they may have about making changes in 
their lives, and resolve that ambivalence. 

It took months of training for advisors to achieve a minimum level of competence in 
motivational interviewing, but the nature and content of family guidance sessions did change 
significantly from Year 1 to the end of Year 2. Participants talked more about their goals, and 
took the lead in creating plans for overcoming barriers to earning rewards. In addition, the 
advisors began conducting targeted outreach to participants with a pattern of earning the fewest 
rewards, a group that proved to be challenging to serve. 
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This change improved the quality of guidance sessions midway through the program, 
but these sessions did not occur in either city as often as envisioned in the model, especially in 
Memphis. Only one-quarter of adult participants met with their advisors more than once in 
Memphis, and most sessions were conducted over the phone. The families’ transportation 
challenges in Memphis greatly impeded the ability of staff to deliver this program component 
effectively. 

• Nearly all families earned rewards from the program, with average 
earnings of just over $2,000 per full program year, or $6,240 for all pro- 
gram years combined. 

Over 98 percent of families in the program group earned at least some rewards during 
the program period, and about 95 percent earned rewards in a full program year. During a full 
program year, the average family earned over $2,000 ($2,300 in Year 2, for example, and 
$2,000 in Year 3). Overall, families earned just about $6,200 in rewards over the full program 
period. 


Families earned the most, on average, from the education rewards. Over 97 percent of 
families earned at least one education reward and the average amount earned was just under 
$3,000 over the full program period. Most families also earned at least one health reward over 
the period (at about 90 percent), and those that did earn health rewards earned on average just 
under $2,000 over the full period. Finally, the work rewards contributed the least to the total 
rewards earned because fewer parents earned them. Only about 57 percent of parents earned a 
work or training reward over the full period, and nearly all of these rewards were for work 
rather than training. 

• Family Rewards 2.0 increased income and reduced poverty during the 
program period and led to improvements in parents’ reports of life sat- 
isfaction and happiness. 

Counting the value of the reward payments, Family Rewards 2.0 increased self-reported 
average household income for the program group by $138 in the month before the 24-month 
survey. (See Table ES.2.) 3 This extra income reduced the proportion of families living at or 
below the federal poverty level by 5 percentage points, compared with the control group rate of 
78 percent. The program also reduced the proportion of families who were living in severe 
poverty (that is, families with incomes less than 50 percent of the federal poverty level) by 7 
percentage points (not shown). All of the poverty reductions from the program are attributable 


’Unless otherwise noted, the impacts cited in this Executive Summary are statistically significant, with less 
than a 10 percent likelihood that they arose by chance. 
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Table ES.2 


Summary of Family Rewards 2.0 Impacts 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Effect 

Size 

Income and poverty 

Average total household income in month prior to 
interview (including Family Rewards payments) 2 ($) 

1,636 

1,498 

138 *** 

0.004 


Household income at or below the federal poverty 
level (including rewards) 2 ’* 5 (%) 

73.5 

78.3 

_4 g ** 

0.012 


Any savings (%) 

20.5 

12.0 

8.5 *** 

0.000 


Children's education 

Graduated on time (%) 

63.4 

63.1 

0.3 

0.855 


Attendance rate is 95% or higher, Year 3 C (%) 

28.9 

29.1 

-0.2 

0.910 


Number of credits earned. Years 1 to 3 d 

30.1 

29.9 

0.2 

0.651 


State core exams passed, Years 1 to 3 e,f 

2.7 

2.7 

0.0 

0.716 


Family health 

Parent had a health checkup (%) 

88.4 

87.6 

0.8 

0.569 


Parent had 2 or more dental checkups (%) 

36.1 

22.5 

13.6 *** 

0.000 


Parent's self-rated health 

(1 = poor; 5 = excellent) 

3.2 

3.0 

0 J H=H=H= 

0.002 

0.126 

Child had health checkup or got shots (%) 

94.9 

94.4 

0.5 

0.614 


Child had 2 or more dental checkups (%) 

62.4 

46.4 

16.0 *** 

0.000 


Parent's work and training 

Has any degree, license, or certificate (%) 

78.5 

77.8 

0.7 

0.651 


Has any trade license or training certification (%) 

46.6 

42.7 

3.9 * 

0.073 


Average quarterly employment, Years 1 to 3 s (%) 

49.6 

52.2 

-2.6 ** 

0.018 


Total earnings, Years 1 to 3 s ($) 

27,684 

29,718 

-2,034 ** 

0.019 


Sample size (total = 2,016) 

1,025 

991 





(continued) 
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Table ES.2 (continued) 


SOURCE: MDRC calculations using data from the Family Rewards 24-month survey, Children's Aid Society's 
Family Rewards program data. New York City Department of Education and Shelby County Schools 
administrative records, New York State unemployment insurance (UI) wage records, and Tennessee Department 
of Labor and Workforce Development UI wage records. 

NOTES: Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics of sample members. Standard errors were adjusted to account for multiple observations per 
family. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The effect size is the difference between program and control group outcomes expressed as a proportion of 
the standard deviation of the outcomes for both groups combined. 

Note that all outcomes in the table include zero values for students who were no longer enrolled. Dollar 
averages include zero values for sample members who were not employed. 

Years 1, 2, and 3 cover the 2011-2012, 2012-2013, and 2013-2014 school years, respectively. 

“Monthly household income amounts equal to or greater than $10,000 were excluded from this 
calculation. About 4.9 percent of the sample was excluded from the income measures because respondents did 
not know or refused to provide the information. An additional 0.2 percent of the sample was excluded because 
the income provided was over $10,000. 

b Annual household income was calculated by multiplying the respondent's income in the month prior to the 
survey interview by 12. For program group members, it includes Family Rewards payments earned during 
program Years 2 and 3. The federal poverty level was calculated based on annual income (monthly income 
multiplied by 12) and the household size at the time of the survey. The poverty threshold was measured 
according to the 2013 or 2014 Poverty Guidelines, depending on when a respondent was interviewed. 

“Attendance was calculated as a percentage of total days present divided by total days enrolled according to 
district records. Records provided for students in New York City include enrollment for the regular school year. 
Records for students in Memphis include enrollment during the regular school year, alternative education 
programs, and summer school. 

d Students in New York City earn 1 credit per course per semester completed. Students in Memphis earn 0.5 
credits per course per semester. Credits for students in Memphis were multiplied by two to create a standard 
scale for comparison. To be considered on time to graduate, students in New York City must earn an average of 
1 1 credits per school year and students in Memphis must earn an average of 5.5 credits per school year. 

“The Regents exam measures in this table include the following Regents exams: English, Math A, Math B, 
Geometry, Integrated Algebra, Algebra 2/Trigonometry, U.S. History and Government, Global History and 
Geography, Living Environment, Chemistry, Physics, and Earth Science. 

r The End-of-Course exam measures in this table include the following exams: English 1, English, 2, English 
3, Biology, Algebra 1, Algebra 2, and U.S. History. 

®This outcome only includes employment and earnings in jobs covered by the New York State and 
Tennessee UI programs. It does not include employment outside of either state, or in jobs not covered by the UI 
system (for example, "off-the -books" jobs and federal government jobs). 
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to the cash transfers that families received, rather than to increased earnings from work. As 
discussed later, the program did not lead to an increase in earnings, and it reduced earnings for 
some families. 

Although the program did increase income and reduce poverty, the effects were fairly 
modest in size, and a fair proportion of the cash transfers were for education rewards, and thus 
deposited in the teenagers’ accounts. As a result, the program did not notably reduce levels of 
material hardship affecting the whole family, such as inability to pay rent or utilities or food 
insecurity. Nonetheless, the extra income families received did lead to improvements in parents’ 
sense of financial security. Parents in the program group, for example, were more likely than 
those in the control group to report that their financial situation was better than it was in the 
previous year and that they could generally afford to buy necessities. Similarly, the program led 
to modest improvements on several measures of psychosocial well-being. For example, parents 
in the program group scored higher than those in the control group on the State of Hope scale, 
which captures respondents’ beliefs in their ability to initiate and sustain action. 

Finally, Family Rewards 2.0 increased families’ connection to the banking system and 
their savings. At the time of the survey, only about 45 percent of parents in the control group 
reported having a savings or checking account, compared with 66 percent of parents in the 
program group, an increase of 21 percentage points (not shown). Very few parents in the control 
group (12 percent) reported having any savings at the time of the survey. The program in- 
creased that rate to 2 1 percent for the program group. 

• The program’s primary effect in the health area was to increase preven- 
tive dental visits, although there is some evidence that it also improved 
adults’ self-reported health status, particularly for those in poorer 
health at study entry. 

Most adults and children in the control group had visited a doctor recently for a routine 
checkup, and the program had no effect on this outcome. In contrast, the majority of both 
parents and children in the control group had not visited a dentist for the recommended two 
dental checkups per year, and the program increased this number substantially. Among children 
in the control group, for example, only 46 percent of them visited a dentist twice during the 
year, compared with 62 percent of those in the program group, an increase of 16 percentage 
points. The importance of oral health has been well documented. 

The program led to a small, positive effect on adults’ self-reported health status, which 
was primarily driven by a sizable effect for those who were in poorer health at study entry. It is 
difficult to know what to make of this effect, given that there were no measured effects on any 
of the other outcomes that might have led to it, such as increased use of preventive care, 
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increased healthful behaviors, and so on. However, Family Rewards 1.0 led to a similar pattern 
of effects on health status for adults in poorer health. 

• The program led to a reduction in employment covered by the UI sys- 
tem, driven largely by reductions in work in Memphis. 

The research team collected and analyzed data on employment from the 24-month 
survey and from UI records, which include all employment covered by the UI system. While 
UI records include data from most employment in the United States, the UI system does not 
cover some jobs, such as self-employment, federal government employment, and domestic 
work. The UI system also misses informal (casual or irregular) jobs that are never reported to 
state agencies. 

The findings show that Family Rewards 2.0 led to reductions in work according to data 
from UI records and had no effect on work reported in the survey. Whereas the reductions in 
Ul-covered work occurred largely in Memphis, in the Bronx the program led to a reduction in 
work reported in the survey. The reason for the difference is unclear. 

Other incentives programs that have focused solely on work have found that increas- 
ing the reward to work can lead to increased employment. 4 However, the work incentives 
here were part of a larger program of rewards. It is likely that the income families received 
from the education and health rewards allowed some parents to reduce work or delay entry 
into the workforce through what is often called an “income effect.” These negative effects on 
work were somewhat larger for parents not working at study entry, and thus more marginally 
connected to work. 

• The program did not affect students’ school progress through Year 4, 
for either the full sample of students or the subgroup of academically 
proficient students. 

The impact analysis examined a range of education outcomes, including attendance, 
grade progression, credits earned, and college entrance exams taken. The program did not have 
effects on these outcomes for the full sample of entering ninth- and tenth-graders. For example, 


4 Nada Eissa and Hilary Hoynes, “Behavioral Responses to Taxes: Lessons from the EITC and Labor Sup- 
ply,” in Tax Policy and the Economy, Volume 20, ed. James M. Poterba (Cambridge, MA: Massachusetts 
Institute of Technology Press, 2006), 74-110; Charles Michalopoulos, Does Making Work Pay Still Pay? (New 
York: MDRC, 2005); Richard Hendra, James Riccio, Richard Dorsett, David H. Greenberg, Genevieve 
Knight, Joan Phillips, Philip K. Robins, Sandra Vegeris, and Johanna Walters, Breaking the Low-Pay, No-Pay 
Cycle: Final Evidence from the UK Employment Retention and Advancement (ERA) Demonstration, UK 
Department for Work and Pensions Research Report No. 765 (Leeds, UK: Corporate Document Service, 
2011 ). 
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63.1 percent of entering ninth- and tenth-graders in the control group had graduated on time 
(that is, within four years of entering ninth grade). The corresponding rate for students in the 
program group was 63.4 percent. When the analysis was focused only on entering ninth- 
graders, the program led to small increases in attendance and credits earned, although it had no 
effects on other outcomes, such as enrollment or graduation. 

The research team also examined impacts for the subgroup of students who scored at 
the proficient level or higher on their eighth-grade English language arts exams, since Family 
Rewards 1.0 had its main effects for this group. Family Rewards 2.0 did not affect education 
outcomes for this subgroup. Finally, subgroup analyses suggested that the program led to a 
small increase in graduation rates among students in less disadvantaged families, which includ- 
ed families in which a parent was working at study entry or that had income above 50 percent of 
the poverty line at study entry. 

Conclusion 

Overall, the results show that the changes to the model, particularly the more intensive guid- 
ance, as it was implemented here, did not make the rewards more effective. Family Rewards 2.0 
did lead to effects in certain domains that were similar to Family Rewards 1.0’s effects, and 
these findings add to the evidence about how a CCT would work in the United States. However, 
the hope was that the new model would have had larger and more widespread effects. In 
particular, the goal was to improve school progress for all students and to help parents move 
into work in order to take advantage of the work rewards. 

There are several possible reasons for the different effects found for Family Rewards 
2.0, compared with those found for Family Rewards 1.0, since the former was not an exact 
replication of the latter. For example, the modified program targeted families receiving TANF 
or SNAP benefits, and it was tested in Memphis. There are also two differences between the 
models to consider. First, Family Rewards 2.0 transferred less money to families than did the 
first program, and there is abundant research suggesting that more income benefits low-income 
children. Perhaps the positive effects found for proficient students in the first program resulted 
from a combination of the rewards acting as incentives and the rewards providing their families 
with more money. Second, Family Rewards 1.0 offered more rewards that families could earn 
regularly, such as school attendance rewards for younger children and rewards for maintaining 
health insurance. While those rewards did not lead to behavioral change in those areas, they 
may have provided a more reliable source of income, which may have contributed to the 
positive effects found for more proficient students. 
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Chapter 1 

Introduction 


Family Rewards was an innovative approach to poverty reduction that was first tested in New 
York City in 2007. Modeled on the conditional cash transfer (CCT) programs common in 
lower- and middle-income countries, the program offered cash assistance to low-income 
families provided that they met certain conditions related to family health care, children’s 
education, and parents’ work. By providing timely cash assistance, CCT programs can reduce 
economic hardship in the short term. By encouraging families to invest in their own develop- 
ment, they have the potential to reduce poverty over the longer term, breaking the intergenera- 
tional cycle of poverty. 

While research has found CCT programs in other countries to have been successful, re- 
ducing poverty and increasing the use of health care services and school enrollment, 1 a compre- 
hensive CCT program had never been tested in a high-income country. Family Rewards was 
sponsored by the New York City Center for Economic Opportunity (CEO), and is one of three 
incentives-based programs that are known collectively as Opportunity NYC. 2 The first iteration 
of the model, or Family Rewards 1.0, was evaluated using a randomized controlled trial, in 
which families were randomly assigned to a program group that was offered the program or a 
control group that was not. While the program did not lead to effects on most of the main 
outcomes, it did lead to notable reductions in poverty while the rewards were being offered and 
increased graduation rates for a group of ninth-graders who were more academically prepared at 
study entry. 3 

The second iteration of the model, or Family Rewards 2.0, is the subject of this report. It 
refined Family Reward 1.0 in several ways: it offered fewer rewards in each of the three areas 
(children’s education, family health, and parents’ work) in order to make the program easier to 
understand and to focus families’ attention on a limited number of outcomes; it offered the 
education rewards only to high school students, given the lack of effects found for younger 
students; it offered students rewards for grades earned; and it made the rewards more timely, 
and thus more salient to families, by paying them monthly for rewards earned, rather than every 


Fiszbein and Schady (2009). 

2 The other two Opportunity NYC programs are Work Rewards, a program offering incentives for work 
and training to recipients of subsidized housing (see Nunez et al., 2015), and Spark, a school-based program 
providing fourth- and seventh-graders with incentives for satisfactory performance or better on tests (see Fryer, 
2011 ). 

3 Riccio et al. (2010); Riccio et al. (2013). 
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two months. Most notably, however, the new model offered guidance, in which staff members 
actively engaged families in conversations about strategies to earn rewards. 

This report examines whether those changes led to bigger impacts and whether the pro- 
gram had similar effects in a context different from New York City. Enrollment for the Family 
Rewards 2.0 study started in July 201 1 in the Bronx, New York, and Memphis, Tennessee. In 
each city, about 1 ,200 families were recruited for the study. Half were randomly assigned to a 
program group that was offered the program, and half were assigned to a control group that was 
not offered the program. Families in the program group could receive rewards from September 
2011 through December 2014, giving all families three years (or a little longer, depending on 
their enrollment date) to earn them. The evaluation tracked participating families during the 
period the rewards were offered to determine the effects of the program on poverty, children’s 
education, family health, and parents’ work. 

An earlier report from the project assessed the implementation of Family Rewards 2.0 
over the first two years, finding that it was operating generally as envisioned in both cities by 
the middle of Year 2, after a few recruitment and start-up challenges. 4 The family guidance 
component evolved considerably over time, becoming much more intensive during Year 2. 
Families also seemed to have understood the program fairly well and earned on average more 
than $2,000 in rewards in Year 2. Those findings suggested that the changes to the model were 
a step in the right direction, leading to a good understanding of the program among participating 
families and higher receipt rates of available rewards. 

In brief, the findings presented here indicate that the new program achieved many of the 
same effects as the original model, but fell short in other, important ways. First, Family Re- 
wards 2.0 met its short-term goals of increasing income and reducing poverty, for all families 
and across a range of family types. The program also increased dental visits and adults’ self- 
reported health status, particularly for those in poorer health at study entry. Similar to the earlier 
program, the new model led to reductions in work and earnings for some participants. However, 
in contrast to the original program, the new model did not affect students’ school progress 
through Year 4, either for the full sample of students or for the subgroup of more academically 
prepared students. 

Funded by the Social Innovation Fund of the Corporation for National and Community 
Service and private foundations, the Family Rewards demonstration was overseen by a collabo- 
rative, which included staff members from The Mayor’s Fund to Advance New York City, 
CEO, and MDRC. The Mayor’s Fund to Advance New York City is a grant-making institution 
that facilitates public -private partnerships throughout New York City. CEO, a unit in the New 


4 Dechausay, Miller, and Quiroz-Becerra (2014). 
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York City Mayor’s Office of Operations, sponsors and manages innovative antipoverty pro- 
grams and evaluations. MDRC worked in close partnership with CEO to design the Family 
Rewards program and led the evaluation. MDRC is a nonpartisan social policy research firm 
with extensive experience conducting large-scale projects using random assignment research 
designs to build rigorous evidence on what works to improve the well-being of low-income 
families. 

The next section presents the results from the Family Rewards 1.0 evaluation. The re- 
mainder of the chapter describes the main elements of Family Rewards 2.0, the model, the two 
cities where it was tested, and the characteristics of the families who enrolled. The chapter 
concludes with an overview of the evaluation. 

Family Rewards 1.0 

Run from 2007 to 2010, the first version of Family Rewards was tested in six of New York 
City’s highest-poverty neighborhoods. Families that were recruited for the study lived in one of 
these neighborhoods, were eligible for the National School Lunch program, and had at least one 
child entering fourth, seventh, or ninth grade. Between July and December 2007, about 4,800 
families enrolled in the study. 

The Theory of Change 

Family Rewards 1 .0 was based on the assumption that, for a variety of reasons, families 
may underinvest in their own development even though such investments can have long-term 
benefits. Financial incentives can help change their calculations while at the same time provide 
cash to help reduce their immediate poverty and hardship. Furthermore, if the rewards are 
sizable, the extra resources can help make it more feasible for low-income people to undertake 
certain educational, health care, and work-related efforts in the short term, by helping them 
cover the cost of, for example, educational materials, tutoring for children, transportation to a 
free dental clinic, or clothes for a job interview. For a more comprehensive discussion of the 
model and its theory of change, see the first Family Rewards 1.0 impact report. 5 

Beyond the direct cash payments to families, the program was expected to reduce cur- 
rent and future poverty through a number of avenues. First, children’s school outcomes were a 
central focus of the program. By providing students with incentives for critical educational 
inputs and outcomes, Family Rewards 1.0 was expected to directly affect their school progress, 
as measured by attendance, grade progression, and test scores. The program designers also 
expected that it might indirectly affect school progress in other ways, such as increased parental 

5 Riccio et al. (2010). 
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engagement, if parents were encouraged to take a more active role in their children’s schooling, 
or by increased family income. It was theorized that increased income may reduce family stress, 
housing instability, and food insecurity, for example, or allow parents to buy educational 
materials for their children . 6 

Similarly, effects on the use of health care services and the family’s health status were 
expected to occur directly through the rewards provided for preventive care, encouraging 
parents and children to obtain regular medical and dental checkups, which may also increase the 
diagnosis and treatment of health problems. However, the designers expected that these rewards 
might also encourage families to maintain health insurance, which would also help them 
maintain a regular family physician, instead of relying on costly emergency room services for 
care once a health problem arises. As with school outcomes, it was thought that the use of 
health care services and the family’s heath status might also improve through increased family 
income, if families had the resources to maintain health insurance, for example, to follow up on 
needed treatments, or had adequate and nutritious food at home. 

The third targeted domain of the program was parents’ work and training. Family Re- 
wards might increase family income through the direct cash transfers themselves, but only 
temporarily. Sustained reductions in child and family poverty after the payments end require 
that parents maintain regular employment. The program thus included a work component 
designed to promote steady, full-time work and the acquisition of skills to help parents qualify 
for better-paying jobs. It was expected that the program should directly affect employment and 
training through these rewards. However, it was thought that the program might indirectly affect 
training if parents had extra income to pay for that training, or if their increased work effort 
highlighted the benefits of additional training. 

Finally, the effects in one domain can reinforce effects in other domains. For instance, 
health care rewards may promote early diagnosis and treatment of certain health and develop- 
mental problems that might otherwise make it difficult for students to sustain high attendance 
and perform well in school. As another example, the work rewards, by increasing parents’ 
earnings, may lead to longer-term effects on family income, which can affect children’s health 
and schooling. 

6 Over the last few years, scientific evidence showing that very low income, material hardship, and finan- 
cial strain have causal influence on children’s life trajectories has grown more convincing. Supporting the 
claim of causal influences are findings from longitudinal studies (Dahl and Lochner, 2012; Duncan and 
Brooks-Gunn, 1997; Gershoff, Aber and Raver, 2003; Mayer, 2002; McLoyd, 1998; Seccombe, 2000) and 
from natural and policy experiments that have effectively raised the income of poor families and evaluated the 
impact of those increases on children (Costello et al., 2003; Morris and Gennetian, 2003; Miller et al., 2008). 
Among the latter studies are evaluations of programs that offered earnings supplements and, in the case of the 
Milwaukee New Hope project, subsidies for child care and health insurance. These evaluations found im- 
provements in young children’s school performance, social behavior, and other developmental outcomes. 
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The Model and Its Effects 

The program offered families 22 cash rewards. In the education area, the program of- 
fered rewards for parents’ attendance at parent-teacher conferences and for children’s attend- 
ance in school, satisfactory performance or better on standardized tests, completion of adequate 
credits per year, passing of exams required to graduate, and graduation. In the health area, it 
offered rewards for maintaining health coverage and for preventive medical and dental check- 
ups for each family member. In the work area, the program offered parents rewards for sus- 
tained full-time work and for pursuing education or training while working. 7 

A final feature of Family Rewards 1.0 is that it offered no services or active counseling. 
Case management and direct services were deliberately excluded from the model in order to test 
the effectiveness of the incentives alone. In addition, it was expected that families in New York 
City would have access to case management and services through other programs in the 
community. 

During the three years the program operated, the average participating family earned 
nearly $9,000 in rewards, or roughly $3,000 in each year. The large majority of families earned 
at least one education reward and one health reward in each year, while just over 40 percent 
earned at least one work reward. The rewards increased families’ incomes by 22 percent on 
average and reduced poverty and severe poverty by about 12 percentage points. However, the 
reductions in poverty and hardship began to diminish after Year 3, when the program ended. 8 

In terms of children’s school progress, Family Rewards 1.0 improved outcomes for 
ninth-graders who were performing relatively better academically when they entered the study, 
with sizable effects on grade completion and graduation. In contrast, the program had no effect 
for lower-performing ninth-graders or for elementary and middle school students. In the health 
area, early positive effects on visits to the doctor and health status faded, in part because 
families already used preventive care at a fairly high rate, although there were continued 
impacts on health coverage and, especially, dental visits. Finally, the program led to modest 
increases in employment throughout the follow-up period, although much of this new work was 
in jobs not captured by the unemployment insurance system. The program also led to a reduc- 
tion in unemployment insurance-covered work for the more disadvantaged adults in the study. 


7 At the end of the second year, in an effort to simplify the program and reduce its costs, several rewards 
were eliminated, including those for maintaining health coverage and for school attendance for elementary and 
middle school students. 

8 See Riccio et al. (2013) and Riccio and Miller (2016) for the full set of findings. 
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Family Rewards 2.0 

In 2010, the Corporation for National and Community Service made its first set of Social 
In novation Fund (SIF) grants. 9 CEO and the Mayor’s Fund to Advance New York City re- 
ceived a grant to replicate and evaluate five of CEO’s most promising antipoverty programs. 
Family Rewards was one of those programs. 

The replication made several important modifications to the original Family Rewards 
model, based on the early experience running the program and its emerging effects. The biggest 
innovation in the design of Family Rewards 2.0 was the addition of the family guidance 
component. Recognizing that many families would need at least some guidance on where they 
could find the kinds of services and assistance that might enhance their success in the program, 
Family Rewards 1.0’s program designers included an infonnation-and-referral component in 
which staff were expected to help educate families about relevant resources that are available in 
the community, through resources guides, marketing materials, and optional workshops. 
However, staff members would generally wait for participants to initiate contact and focus on 
issues directly related to earning rewards rather than reaching out to participants early on and 
walking them through potential strategies. 

Based on findings from Family Rewards 1.0, the program designers believed that fami- 
lies would benefit from a regular source of support within the program. The new model would 
use a triage model to provide this guidance, in which all participants would receive some 
guidance, but those who were having the most trouble earning rewards would receive more 
intensive outreach and support, with staff connecting them to the resources and help they 
needed to earn rewards. Findings from Family Rewards 1.0 suggested that there were several 
types of participants who might have benefited from this additional support, such as students 
who started high school scoring below grade level on eighth-grade math and English exams 
(“non-proficient” students). 

Program Delivery 

A consortium ran the demonstration that included The Mayor’s Fund to Advance New 
York City, CEO, and MDRC. This consortium set the broader design policies and provided 
technical assistance and oversight, largely through the lead program operator. Seedco, a New 


9 The SIF was an initiative enacted under the Edward M. Kennedy Serve America Act and designed to 
create and fund a catalog of proven policies that can be replicated in communities across the country. For this 
purpose, the SIF uses federal funds to raise additional support for evidence-based solutions to pressing social 
issues. 
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York-based nonprofit organization, served as the lead program operator until summer 2012, 
when Children’s Aid Society took over the role. 10 

Four Neighborhood Partner Organizations (NPOs) were selected to operate the pro- 
gram. The NPOs were selected based on the experience they had in program delivery in their 
local communities. In the Bronx, the NPOs were the Children’s Aid Society and Bronx- 
Works. In Memphis, the NPOs were Urban Strategies Memphis HOPE (Memphis HOPE) 
and Porter-Leath. 

All NPOs were charged with implementing the program’s core components, such as re- 
cruitment, outreach, and family guidance. Each NPO had its own Family Rewards supervisor, 
who was responsible for multiple programs at the NPO, including Family Rewards. The 
supervisor directly managed three advisors, which allowed each NPO some level of autonomy 
in delivering the program elements. The advisors were charged with providing guidance to 
families about how to earn rewards. 

The Rewards 

Table 1.1 presents the rewards for Family Rewards 2.0. (See Appendix Table A.l for a 
comparison of rewards in Family Rewards 1.0 and 2.0.) The top panel shows the education 
domain, in which students earned rewards for high attendance, good grades, satisfactory 
performance or better on state core exams, and taking college entrance exams. One addition to 
the list of rewards in the new program was a reward for grades, in an effort to make the rewards 
more timely and more strongly tied to daily performance. In addition, students in Family 
Rewards 2.0 received all the education rewards they earned, deposited directly in their bank 
accounts, in contrast to Family Rewards 1.0 in which parents received a portion of those 
rewards. The second panel shows rewards for health, in which families received payments for 
medical and dental checkups for each family member. The new program did not include the 
reward for maintaining health insurance, since the original study found that most families 
already had coverage. Finally, the program rewarded parents for maintaining full-time work and 
for earning a General Educational Development (GED) certificate. The original model rewarded 
a range of approved training courses, but few families received the training reward. For this 
reason, the new program focused on one outcome, the GED certificate, which was well known 
to participants and could serve as a stepping-stone to further training. 


10 CEO transferred overall management of SIF Family Rewards to Children’s Aid Society in 2012 after 
local and federal investigations uncovered irregularities in Seedco’s administration of workforce services 
unrelated to Family Rewards. 
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Table 1.1 


Family Rewards 2.0 Schedule of Rewards 


Domain 

Bronx 

Memphis 

Education incentives 
(high school students only) 



Attends 95% of scheduled 
school days 

$40 per month 

$40 per month 

Takes an SAT or ACT exam 
(once during program) 

$50 

$50; must score 19 or more 
on the ACT if administered 
by Memphis City Schools 3 

Receives grades on an official 
report card 1 ’ 

$30 per A (90-100); $20 per 

B (80-89); $10 per C (75-79) 

$30 per A (93-100); $20 per 

B (85-92); $10 per C (75-84) 

Passes up to 5 Regents exams 0 
or 7 End-of-Course exams d 

$500 per Regents exam for a 
score of 75 or above; $400 
per exam for a score of 65-74 

$200 per End-of-Course 
exam for a score of proficient 
or advanced (increased to 
$300 in Year 3) 

Health incentives (parents and 
children 19 years and younger) 



Annual medical checkup 

$100 per family member 

$100 per family member 

Preventive dental care every 

6 months (once per year for 
children 1 to 5 years) 

$100 per family member 

$ 1 00 per family member 

Work (parents only) 



Sustains full-time employment 

$150 per month 

$150 per month 

Earns a GED certificate 

$400 

$400 


SOURCE: Children’s Aid Society’s Family Rewards program materials. 

NOTES: GED = General Educational Development. 

The ACT is scored out of 36. Memphis City Schools officials requested that the minimum score for 
the reward be set at 19 for students taking the test for free in class because this score is considered an 
indication that students are ready for college-level work. 

b In Year 1, the amount of each reward for grades was prorated based on the number of official report 
cards issued by a student's school. To simplify verification in Year 2, students were paid the listed amounts 
for their grades regardless of the number of official report cards they received, up to a maximum of $600 
per program year. 
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Table 1.1 (continued) 


c High school students (grades 9-12) in the Bronx were eligible to earn rewards for the following Regents 
exams: English, one of any math exams (including Math A, Math B, Integrated Algebra, Geometry, and 
Algebra 2/Trigonometry), U.S. History and Government, Global History and Geography, and one of any 
science exams (including Living Environment, Chemistry, Physics, and Earth Science). 

d High school students (grades 9-12) in Memphis were eligible to earn rewards for the following End-of- 
Course exams: Algebra 1, Algebra 2, Biology, English 1, English 2, English 3, and U.S. History. 

As in Family Rewards 1.0, families in Family Rewards 2.0 received payments for meet- 
ing reward milestones in one of two ways. The first required no action on the part of the family 
and applied to two of the rewards, for attendance and for performance on state core exams, 
which were automatically verified using school records. The second applied to all other rewards 
and required the family to actively claim the reward by submitting a “coupon,” verifying that 
the milestone has been met, to the program. For example, to claim the reward for full-time work 
the family had to submit a coupon along with other items of verification. 

In an effort to make the rewards more timely and thus more salient, families in Family 
Rewards 2.0 received payments for most rewards every month, rather than every two months as 
did those in the original program, along with an earnings statement documenting earnings for 
the month and any rewards that the family was eligible for but did not earn. Finally, families 
were strongly encouraged to open low-fee bank accounts in order to take advantage of the direct 
deposit of rewards. To facilitate this process, the program invited banks that offered no-fee, 
savings accounts to attend orientation sessions and market directly to families. 

Family Guidance 

Family Rewards 2.0 offered participants active and tailored guidance from designated 
staff members, or advisors. In the new model, advisors at the NPOs developed a Family Earning 
Plan with every family and met with them at least twice a year to discuss their progress toward 
earning the rewards. Advisors also proactively reached out to and engaged those families who 
were not earning many rewards. Staff also had access to a small resource fund, to help families 
with services they might need to earn rewards, such as short-term tutoring, uniforms for work, 
transportation to job interviews, or licensing fees. As discussed in the implementation report, the 
strategy for family guidance shifted during Year 2, as staff members were trained in the use of 
motivational interviewing, a client-centered counseling method that has been found in other 
settings to be effective at eliciting behavioral change. 

As in Family Rewards 1.0, the theory of change underlying the new model is that the 
rewards, in combination with the guidance, will lead to a number of effects. Figure 1.1 presents 
a simplified depiction of the overall logic model, showing the main processes through which the 
rewards and guidance were expected to reduce current and future poverty. 
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Figure 1.1 
Theory of Change 









Target Groups 

Family Rewards 2.0 targeted families with at least one child entering ninth or tenth 
grade. Although it only offered the education rewards to students who were ninth- or tenth- 
graders at study entry, once enrolled in the program, all of the family’s children under 20 years 
of age became eligible for the health-related rewards. The program also targeted families 
receiving Temporary Assistance for Needy Families (TANF) or Supplemental Nutritional 
Assistance Program (SNAP, or “food stamps”) benefits, in order to direct resources to the 
neediest families and to explore how a CCT approach might interact with and complement 
these safety net programs. 

The Places 

Family Rewards 2.0 was tested in the Bronx, New York, and Memphis, Tennessee. 
The idea behind including another city in the replication effort was to make the findings, 
across both cities together, more relevant to the broader national population. But each city 
also contributed to the analysis, individually. By running the program again in New York 
City, the program designers expected the evaluation could answer whether the changed model 
works better in a similar context. By adding Memphis, they anticipated that the evaluation 
could also assess how the model would work outside of New York City, in a very different 
economic and social context. In this respect, Memphis represented a good addition to the 
evaluation. With its smaller population (700,000 residents in Memphis versus 1.4 million in 
the Bronx) and different racial and ethnic makeup, Memphis provided a very different context 
from the Bronx. Income levels and poverty rates were roughly similar in the two areas in 
early 2013, although the labor markets were fairly different. The unemployment rate was also 
roughly similar in both areas in 2013. 11 Median wages were much lower in Memphis (about 
$15 per hour) than they were in the New York City area ($21 per hour), although the cost of 
living is also much lower in Memphis. 12 

Finally, as noted in the implementation report, there were several differences between 
the two cities that might have affected program implementation and impacts. First, the state- 
sponsored health insurance program for low-income New Yorkers covered routine dental care, 
while the similar plan in Memphis (TennCare) did not. Dental care coverage may have affected 
the ability to earn the dental reward. Second, families and students in the Bronx had access to an 
extensive public transportation system, whereas families in Memphis relied heavily on cars. The 


n U.S. Department of Labor, Bureau of Labor Statistics (2014). 

12 U.S. Department of Labor, Bureau of Labor Statistics (2015a and 2015b). While this difference in medi- 
an wages partly reflects the different mix of occupations in the two areas, median wages are also lower in 
Memphis for specific occupations, such as “home health aide” jobs or jobs in food preparation and service. 
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lack of quality public transportation made study recruitment more difficult in Memphis and also 
affected how families participated in the program. Finally, whereas most of the organizations 
and resources listed in the resource guides provided by the staff at the Bronx-based NPOs were 
located in the Bronx, a deliberate decision on their part, most of the organizations and resources 
listed in the Memphis guides were located in the downtown or midtown area, not close to where 
participants lived. Transportation challenges may have kept Memphis participants from access- 
ing the services that might have helped them earn rewards. All of these differences suggest that 
Bronx residents had easier access to the resources that could help them earn rewards. 

Program Duration 

The program was intended to be offered to all participating families for three years, 
starting in September 2011 and ending in August 2014. Because some families entered the 
study after September 2011, the program designers extended the program period beyond three 
years, to December 2014, in order to offer these late-entering families three years of rewards. 
Thus, all families had access to at least three years of rewards, and families who entered the 
study earlier had access to rewards for somewhat longer than three years. Ninth-graders who 
entered the study in August 2011, for example, were eligible for rewards through the fall 
semester of their senior year, assuming they graduated to the next grade every year. 


Sample Characteristics 

Recruitment began in August 2011 in the Bronx and September 2011 in Memphis, when the 
NPOs received lists of potentially eligible participants provided by the human services agencies 
and departments of education in each city. 13 NPO staff members were then required to call 
families on the lists, making multiple attempts to reach eligible families, including additional 
phone calls and home visits. While enrollment was originally expected to conclude by October 

2011, staff faced several challenges to meeting enrollment targets, and as a result the majority of 
families in the study enrolled after October, and many enrolled as late as January or February 

2012 . 


In total, Bronx NPOs enrolled 1,232 households and Memphis NPOs enrolled 1,226 
households. Tables 1.2, 1.3, and 1.4 present selected characteristics of the samples. There were 


13 In the Bronx, the recruitment lists provided by the New York City Department of Education included 
information on families’ receipt of TANF and food stamps. In Memphis, the recruitment lists did not include 
benefit status. Thus, program staff in Memphis determined whether enrollees were eligible for Family Rewards 
based on self-reported receipt of TANF or food stamps. 
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Table 1.2 


Selected Household Characteristics at Enrollment, by City 


Characteristic 

Bronx 

Memphis 

One-parent family 3 (%) 

77.1 

90.5 

Marital status (%) 

Single 

49.8 

70.8 

Married 

20.9 

8.0 

Separated/divorced/widowed 

27.2 

19.7 

Living with partner or legal domestic partner 

2.0 

1.5 

Average number of children in household (under 19 years old) 

2.4 

3.0 

Primary language (%) 

English 

48.2 

99.1 

Spanish 

49.8 

0.6 

Currently receiving food stamps (%) 

99.4 

98.0 

Currently receiving TANF or Safety Net Assistance (%) 

5.0 

30.5 

Living in rental housing (%) 

94.3 

71.6 

Living in public housing (%) 

34.6 

10.3 

Receiving Section 8 rental assistance (%) 

23.5 

13.6 

During the last 12 months, ever unable to pay (%): 

Rent or utility bills 

49.3 

64.0 

Telephone bills 

30.4 

32.8 

Food or prescription drug costs 

25.1 

24.2 

Sample size 

1,230 

1,226 


SOURCES: MDRC calculations from baseline survey and random assignment module data. 

NOTES: Percentages for some categories may not add up to 100 due to rounding or missing values. 

a This measure includes families with parents who reported their marital status as single, single but 
living with a boyfriend or girlfriend, separated, divorced, or widowed. 


some differences between the city samples. As shown in Table 1.2, for example, a larger 
fraction of participating families were headed by a single parent in Memphis (90 percent) than 
those in the Bronx (77 percent). A much larger fraction of the sample in the Bronx was Hispan- 
ic than the sample in Memphis and more sample members in the Bronx spoke Spanish as their 
primary language than those in Memphis. In contrast, nearly all participating families in 
Memphis were African American. Probably reflecting the tighter housing market in New York 
City, more families in the Bronx lived in subsidized housing when they entered the study: 35 
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Table 1.3 


Selected Adult Characteristics at Enrollment, by City 


Characteristic 

Bronx 

Memphis 

Demographic characteristics 

Female (%) 

87.6 

93.9 

Average age (years) 

42.6 

39.3 

Race/ethnicity (%) 

Hispanic 

74.4 

0.9 

African American 

22.9 

98.1 

Other 

2.7 

1.0 

Citizenship status (%) 

U.S. citizen by birth 

42.8 

98.6 

U.S. citizen by naturalization 

31.7 

0.8 

Legal permanent resident 

25.5 

0.6 

Educational characteristics (%) 

Highest degree received 

High school equivalency certificate 

6.2 

7.6 

High school diploma 

32.8 

51.2 

Technical/associate's/2-year college degree 

7.3 

6.2 

4-year college degree or higher 

5.9 

4.0 

None of the above 

47.9 

31.0 

Trade license or certificate 

45.0 

27.6 

Employment characteristics 

Currently employed (%) 

57.1 

44.1 

Full time (at least 30 hours per week) 

39.4 

34.0 

Part time 

17.6 

9.9 

Among adults currently employed 

Average weekly wage ($) 

314 

332 

Has a physical problem that limits work (%) 

16.0 

18.3 

Has an emotional or mental health problem that limits work (%) 

6.5 

7.9 

Receives SSI or SSD1 due to disability (%) 

13.1 

15.1 


(continued) 
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Table 1.3 (continued) 


Characteristic 

Bronx 

Memphis 

Medical coverage and care (%) 

Source of health insurance coverage 

Public health insurance 

85.9 

81.9 

Employer-provided or other health insurance 

10.3 

10.1 

No coverage 

3.8 

8.0 

Usual source of medical care 

Clinic or health center 

73.2 

49.7 

Doctor's office or health maintenance organization 

16.1 

41.0 

Hospital emergency room 

3.3 

7.8 

Hospital outpatient department 

7.2 

1.0 

Other 

0.2 

0.5 

Date of last medical checkup 

Less than 12 months ago 

87.4 

76.5 

12 months ago or longer or never had a checkup 

12.6 

23.5 

Date of last dental checkup 

Less than 12 months ago 

77.8 

35.7 

12 months ago or longer or never had a checkup 

22.2 

64.3 

Health status (%) 

General health status 

Excellent/very good 

35.3 

35.8 

Good 

38.0 

38.4 

Fair/poor 

26.7 

25.8 

Felt down, depressed, or hopeless during the past 2 weeks 

22.1 

23.3 

Financial characteristics (%) 

Has a bank or credit union account 

59.2 

39.7 

Sample size 

1,311 

1,256 


SOURCES: MDRC calculations from baseline survey and random assignment module data. 

NOTES: Percentages for some categories may not add up to 100 due to rounding or missing values. 
Italics indicate measures calculated for a subset of the full sample. 

SSI = Supplemental Security Income. SSDI = Social Security Disability Insurance. 
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Table 1.4 


Selected Child Characteristics at Enrollment, by City 


Characteristic 

Bronx 

Memphis 

Demographic characteristics 

Female (%) 

50.4 

51.6 

Age (%) 

0 to 5 years 

10.8 

9.6 

6 to 13 years 

31.0 

36.8 

14 years or older 

58.2 

53.6 

Race/ethnicity (%) 

Hispanic 

72.7 

1.0 

African American 

23.8 

97.7 

Other 

3.5 

1.4 

Born in the United States (%) 

87.3 

99.6 

Educational characteristics (%) 

Expected grade level in September 2011 

Preschool 

10.0 

9.7 

Grades 1 to 5 

17.3 

19.7 

Grades 6 to 8 

13.1 

18.6 

Grades 9 to 12 

54.3 

50.4 

Not enrolled 

1.4 

0.9 

Enrolled as an English Language Learner in the previous school year 

21.6 

5.5 

Had an Individualized Education Program in the previous school year 

16.3 

13.8 

Received tutoring help in the previous school year 

26.5 

30.0 

Nonproficient on English exam 3 

75.2 

85.6 


(continued) 


percent of families in the Bronx reported living in public housing, compared with only 10 
percent of families in Memphis. In contrast, the number of families receiving TANF benefits 
was much higher in Memphis, at 3 1 percent, than the number in the Bronx, at 5 percent. The 
last rows of the table illustrate the material hardships faced by enrolled families. More than half 
of families reported being unable to pay a rent or utility bill within the previous year, and a 
quarter were unable to pay for food or needed medicine. 

Table 1.3 shows that enrolled adults in the Bronx had somewhat lower education lev- 
els than those in Memphis, reflecting in part the fact that more of the enrolled adults in the 
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Table 1.4 (continued) 


Characteristic 

Bronx 

Memphis 

Medical coverage and care (%) 



Has public health insurance 

90.2 

95.7 

Has employer-provided or other health insurance 

8.0 

3.8 

Usual source of medical care 



Clinic or health center 

79.8 

55.3 

Doctor's office or health maintenance organization 

14.3 

41.3 

Other 

5.9 

3.4 

Date of last medical checkup 



Less than 12 months ago 

94.2 

88.2 

12 months ago or longer or never had a checkup 

5.8 

11.8 

Date of last dental checkup 



Less than 12 months ago 

85.8 

86.3 

12 months ago or longer or never had a checkup 

14.2 

13.7 

Health status (%) 



General health status 



Excellent/very good 

73.5 

77.4 

Good 

23.5 

19.9 

Fair/poor 

3.0 

2.8 

Has a physical problem that limits activities 

4.7 

5.4 

Has an emotional or mental health problem that limits activities 

6.0 

5.5 

Sample size 

2,914 

3,408 


SOURCES: MDRC calculations from baseline survey data and random assignment module data 
and New York City Department of Education and Shelby County Schools administrative 
records. 

NOTES: Percentages for some categories may not add up to 100 due to rounding or missing 
values. 

a Proficiency level is only reported for high school students who had taken a standardized test 
to determine proficiency within the two years prior to enrollment. Data were available for most 
students who were in ninth or tenth grade at enrollment. 


Bronx were immigrants. 14 For example, nearly half of the adults in the Bronx sample did not 
have a high school diploma or equivalency certificate at the time of study enrollment, com- 
pared with only 3 1 percent of adults in Memphis. In contrast, adults in the Bronx sample were 
more likely to be working at the time of study entry (57 percent) than their counterparts in 


14 Foreign-bom study participants in the Bronx had lower education levels than U.S.-born participants. 
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Memphis (44 percent). This difference is also related to immigrant status, given that foreign- 
bom study participants in the Bronx sample were more likely to be working than their U.S.- 
bom counterparts. 

Finally, Table 1.4 presents selected characteristics of the children in participating fami- 
lies, including the target ninth- and tenth-graders and their siblings. Other than race and ethnici- 
ty and English-language learner status, there were no large differences between the city sam- 
ples. Most children had health coverage at study entry, about 15 percent of children had 
participated in an Individualized Education Program (IEP), 15 and most parents reported that 
their children were in excellent or good health. 


The Evaluation 

The evaluation included a comprehensive study of Family Rewards 2.0, consisting of three 
components. 16 

1. Implementation analysis. This component includes a description and assessment of 
how families were recruited into the study and how the CCT incentives were marketed and 
communicated to families. This analysis, most of which was presented in the earlier implemen- 
tation report, examines, for example, how parents and youth understood the Family Rewards 
cash incentives, how many rewards families received, what factors were associated with reward 
receipt, and what strategies the NPOs used to engage families. Because of the extended study 
enrollment period, the first program year (September 2011 through August 2012) was a partial 
year for families who enrolled later in the period. Similarly, the final program period (Septem- 
ber 2014 through December 2014) was only four months long. For this reason, the analysis of 
reward receipt focuses on Years 2 and 3, when all families had had a full year to earn rewards. 

Data for this analysis include observations of program activities (including family guid- 
ance sessions), interviews with advisors at the NPOs and Children’s Aid Society, focus groups 
with adult and high school student participants, a review of all program materials, a case-file 
review to analyze the implementation of the Family Earning Plans, and an analysis of the 
management information system and payment-processing data. 

2. Impact analysis. This component examines the program’s effects on a wide range of 
outcomes. The analysis assesses whether Family Rewards 2.0 increased family income and 


15 Mandated by the Individuals with Disabilities Education Act, IEPs are intended to help children with 
disabilities achieve their educational goals. The percentage of students participated in an IEP was similar for 
the high school students and their younger siblings in the sample. 

16 Appendix E presents more information about the data sources and method used for each of the three 
study components and is available on the MDRC website at www.mdrc.org. 
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reduced material hardship, and whether the program affected a range of outcomes both directly 
and indirectly related to the reward milestones, such as children’s performance in school, health 
care use and outcomes, and parents’ employment and earnings and participation in training. The 
main analysis focuses on effects across the two cities combined, in order to provide a more 
general assessment of the effects. However, the report also examines whether the program’s 
effects differed between the two cities and for different types of families. 

Data sources include administrative records on students’ school performance, parents’ 
employment and earnings, and families’ receipt of benefits. A survey was administered to 
families 24 months after study entry, capturing program experiences, income and material 
hardship, and other outcomes. 17 Finally, depending on the data source, a family’s “follow-up” 
year does not always align with the “program year.” The education data, for example, were 
aligned with the program year, since the school year began in September. Thus, for those who 
entered the study after September, Year 1 of education data includes several months before they 
were eligible for rewards. Impacts on education outcomes in Year 1 should be interpreted with 
this factor in mind. However, the follow-up period for all other data sources, such as unem- 
ployment insurance-covered earnings, started once a family entered the study, so that Year 1 
outcomes for these sources reflected a full year of program participation. 

3. Cost analysis. This component estimates the cost to administer rewards and the cost 
of reward payments to participants. Costs are estimated for each reward type. Cost data sources 
include the NPOs’ budgets and expenditure reports as well as reward payment records. The 
analysis shows how much money was transferred to participants in the fonn of rewards as well 
as how much it cost to administer the transfers. The analysis also explores the potential benefits 
to participants and how the program stands up to a benefit-cost test. 

Although this report’s main goal is to present the effects of Family Rewards 2.0 after 
three to four years, comparisons are made throughout between these effects and those of Family 
Rewards 1.0. However, this comparison is not one of “apples to apples.” First, the two pro- 
grams served different types of families. For example, Family Rewards 2.0 targeted families 
receiving TANF or SNAP benefits, whereas the original program targeted families eligible for 
the free lunch program. Families in the latter group therefore earned on average higher incomes. 
Second, the programs were tested in different places. Family Rewards 2.0 operated in the Bronx 
and Memphis, whereas the original program operated only in New York City neighborhoods 
(the Bronx, Brooklyn, and Manhattan). A more direct assessment of the new program’s added 
value would focus on effects for the same types of families in the same neighborhoods. 


l7 The 24-month survey achieved response rates of 81 percent for parents in the control group and 83 per- 
cent of parents in the program group. Appendix F includes a detailed analysis of survey response and is 
available on the MDRC website at www.mdrc.org. 
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Organization of This Report 

The remainder of the report is organized as follows. Chapter 2 discusses the implementation of 
the program across both cities and presents estimates of its costs. Chapters 3 through 6 present 
the program’s effects in each of the main domains, for both cities combined. Chapter 3 focuses 
on the effects on income and poverty, and Chapters 4 through 6 present the effects on each of 
the three domains of education, health, and work. Chapter 7 presents effects for selected 
subgroups and for each city separately. Chapter 8 summarizes the report’s findings and discuss- 
es some broad lessons from both Family Rewards programs. 
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Chapter 2 

Program Implementation and Reward Payments 


MDRC’s earlier report on Family Rewards 2.0 described the implementation of the program 
during its first two years of operation, from September 1, 2011, to August 31, 2013. 1 The 
program continued to offer rewards and guidance into a partial fourth year, through December 
2014. This chapter provides an update to that analysis, describing the delivery of the program 
during the final 18 months and how much money families earned in rewards overall. It gives 
special attention to the way the guidance component was implemented, as it was the biggest 
change to the program design compared with the earlier version. 

Providing tailored, proactive assistance to each adult and high school student participant 
introduced a higher level of operational complexity to the conditional cash transfer (CCT) 
program, and increased the cost of delivering the program relative to Family Rewards 1.0. 2 In 
the second year of the program, significant program resources were devoted to training desig- 
nated staff, or advisors, and developing an approach to providing guidance that incorporated 
motivational interviewing (MI) techniques. The earlier implementation report found that this 
shift improved the quality of advisor-participant interactions, although guidance sessions neither 
occurred as often as called for by the model’s design (at least twice per year), nor were they 
consistently targeted to those earning the fewest rewards. Still, the earlier report predicted that 
participants would reap the benefit of these investments in program improvement in Years 3 
and 4 in the form of higher reward earnings — especially in challenging categories such as adult 
employment — since all of the program’s components were fully operational at that time. This 
chapter examines whether that prediction bore out. 


Dechausay, Miller, and Quiroz-Beccera (2014). 

2 Few CCT programs around the world have attempted to combine individualized guidance with incen- 
tives. While there is broad recognition that families often face barriers to accessing quality services, it is more 
common to address this problem by facilitating peer support rather than pairing families with trained counse- 
lors. Two exceptions to this trend are the programs Solidario in Chile and Bolsa Familia in Brazil. In the 
Chilean model, social workers visit families frequently during the first 6 months after enrollment and intermit- 
tently for an additional 18 months (Calvo, 2011). They provide psychological support and facilitate preferential 
access to services in an effort to help families complete 53 conditions and graduate successfully from the 
program. Social workers in the Brazilian model perform the dual role of verifying that families are meeting the 
conditions for transfers and helping families overcome barriers. They can recommend that a family be dropped 
from the program for not meeting conditions (Soares et al., 2009). 
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Summary of Implementation Findings 

The implementation analysis of Family Rewards 2.0 covered both cities and was partially 
conducted by Spanish-speaking researchers since some of the program activities in the Bronx 
took place in Spanish. The data collected included: (1) observations of program activities 
including guidance sessions and review of materials throughout the program; (2) annual 
interviews with family advisors at the NPOs and staff from Children’s Aid Society’s central 
office in Manhattan, New York (CAS-Central), which managed the overall program; (3) focus 
groups with 85 adults and 57 high school students who were members of subgroups of interest 
(for example, students who scored proficient or better on their eighth-grade reading exams and 
those who did not) or who earned a certain number of rewards in Years 1 and 2; (3) interviews 
with 2 1 adults and 1 8 high school students who were a mix of high and low reward-earners in 
Years 3 and 4; (4) analysis of all payment and case management data from CAS-Central’s 
management information system (MIS); and (5) participation and engagement items on the 24- 
month survey delivered only to program group members. The analysis of the costs of the 
program was based on data from the CAS-Central’s financial accounting system. 

The analysis found that the guidance component evolved during the program, becoming 
more intense (focused on barrier identification rather than issues related to the mechanics of 
claiming rewards) after MI was adopted in Year 2. This shift improved the quality of guidance 
sessions midway through the program, though these sessions did not occur as often as envi- 
sioned in the model. In Year 3, strong distinctions emerged in the way that NPOs in the two 
cities delivered the guidance component. Advisors at NPOs in the Bronx found ways to increase 
the frequency and intensity of their interactions with participants, whereas in Memphis only 
one-quarter of adult participants met with their advisors more than once in Year 3 and most 
sessions were conducted over the phone. The transportation challenges in Memphis greatly 
impeded the ability of NPOs to deliver this program component effectively. 

The payment processing system was well implemented and parents described it as effi- 
cient and accurate. On the other hand, many parents submitted coupons for rewards that were 
rejected, and reported a sense of discouragement with claiming certain incentives. 3 Most 
enrolled families (99 percent) earned at least one reward during the program. These families 
received an average of $6,241 overall, about half of which was transferred directly to high 
school students’ bank accounts for education rewards. Most families earned rewards in educa- 
tion and health, but only about 57 percent ever earned a work reward. 


3 As noted in the earlier implementation report, coupons for report card grades and full-time work were 
rejected the most frequently, often because program participants submitted inadequate documentation. 
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The pattern of reward receipt across the two cities was fairly similar, although families 
in Memphis received an average of about $950 less than families in the Bronx. Over approxi- 
mately three years, the program spent $13,459 on the typical participating family. Nearly half of 
this amount (48.3 percent) was for rewards paid. 

The next section highlights the main aspects of the program’s operations. It is followed 
by more detailed descriptions of reward receipt and the guidance component and a benefit-cost 
analysis. 


The System for Processing Payments 

The CAS-Central payment processing system ensured that participants received their earned 
incentive payments in a timely fashion. As noted in Chapter 1, Family Rewards 2.0 offered 
eight rewards in three areas: education, health, and work. (See Table 1.1.) While Family 
Rewards 1.0 offered most of these same rewards, the new program sought to make these 
rewards easier to understand and claim for enrolled families. For example, program designers 
changed the criterion for earning the full-time work reward from demonstrating 30 hours of 
work per week to showing 120 hours of work per month. The reward for high school students’ 
grades was new and turned out to be very difficult to administer. 4 

Children’s Aid Society operated a central office in Manhattan, New York (CAS- 
Central), for all participants in the evaluation. CAS-Central received documents from partici- 
pants and administrative records from the school districts to verily completion of activities, 
determined whether participants were eligible for payments, and deposited payments in their 
bank or stored value card accounts. Staff who reviewed coupons also answered the program’s 
toll-free helpline, which parents, high school students, and advisors could call with questions 
about paperwork, bank account issues, or payments. 

CAS-Central paid families every month for rewards earned about six weeks earlier. The 
lag between earning the reward and receiving payment was due to the time it took to manually 
verify eligibility for payments. The staff provided earnings statements in English and Spanish 
for each family that explained the earned rewards for which the family had been paid, as well as 
the rewards the family had not earned. CAS-Central staff created these statements manually and 
thus mailed them three to four weeks after payments were posted to participants’ accounts. 
Program designers had hoped payments would be received closer to the conditioned behavior to 
amplify the reinforcing effects of the incentives, but that was not achieved because the demon- 
stration’s small scale precluded fully automating verification procedures. 


4 See Dechausay, Miller, and Quiroz-Beccera (2014) for more information. 
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The 24-month survey investigated parents’ understanding of the conditions to earn re- 
wards and their experiences with the payment processing system. Generally, parents demon- 
strated a strong understanding of the behaviors that were eligible for rewards and the cash value 
of the rewards, although many believed the program rewarded additional behaviors that were 
not eligible for rewards. (See Appendix Table B.l.) The only reward about which there was 
widespread confusion among participants was the one for proficient or above perfonnance on 
the Tennessee End-of-Course exams for high school students. While 94.8 percent of parents in 
the Bronx reported that their children knew the numeric score on the equivalent New York State 
Regents exams they needed to obtain to qualify for the reward, only 19.7 percent of parents in 
Memphis reported that their children understood the level of perfonnance they needed to 
demonstrate on the End-of-Course exams to qualify for it. (Students needed to receive a score of 
“proficient” or above to earn the reward.) The End-of-Course exams in Memphis were new at 
the time Family Rewards 2.0 was launched, and students’ raw scores were scaled to determine 
proficiency. The survey findings suggest that either advisors did not succeed in explaining this 
reward to high school students in Memphis by the time of the survey, or the inherent uncertainty 
of this perfomiance standard made it difficult for families to understand. 

Parents gave the payment processing system strongly positive reviews on the survey, 
with about 81 percent agreeing with the statement that they received payments on time and 91 
percent reporting that they understood the reasons when they did not receive a reward or 
payment for a behavior. The majority of survey respondents (about 88 percent) also reported 
that they “remember the deadline for coupons and get them filled out completely.” However, 
despite this strong positive response among survey respondents, most families enrolled in the 
program had coupons rejected in Year 2. An MDRC analysis found that 90 percent of Memphis 
families and 62 percent of Bronx families had at least one rejected coupon that had not been 
resolved by the end of the year. 5 In fact, almost every participant who submitted a coupon for 
full-time work had at least one rejected. There is evidence that these rejections led to discour- 
agement. Forty-two percent of survey respondents reported that they had given up on earning 
rewards because “it is hard to get paid for things I do.” As discussed below, most families 
earned rewards throughout the program, but this finding suggests that individuals within 
families may have given up on earning particular rewards entirely or for a period of time due to 
challenges they experienced with the paperwork or criteria. 


5 In this case, “resolved” means that the error was addressed, the family or the staff member resubmitted 
the coupon, and the coupon was approved. 
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The Rewards Families Earned 


Table 2.1 presents a summary of rewards earned by families in the Bronx and Memphis in all 
four program years. Years 1 and 4 were partial years for most participants, while Years 2 and 3 
were full years and can be directly compared. The table shows that 99 percent of families 
earned at least one reward, and they earned an average of 100 rewards. The high number of 
rewards is due to those offered for report cards, in which each qualifying grade was counted 
separately. In Years 2 and 3, families earned an average of just over $2,000 per year from 
rewards ($2,314 and $2,039, respectively), with about half of this amount paid to high school 
students for education rewards. Reward receipt was highest in the second year of the program, 
largely because of the high number of students who earned the high-value rewards for state 
exams (about 67 percent). 

To put these annual earnings in context, the national average earned income tax credit 
(EITC) received for the 2014 tax year was $2, 335. 6 The EITC has been described as “one of the 
nation’s most effective anti-poverty programs.” 7 The EITC and Family Rewards incentives 
differ in their certainty, salience, and beneficiaries. Family Rewards was less certain or predict- 
able because the amount earned depended on which milestones families reached (and whether 
they made a proper claim for payment). For some milestones, particularly in education, there 
was a high performance standard. The reward money earned from Family Rewards was also, 
arguably, less salient because it was distributed each month in different amounts and intermin- 
gled with other money in the participant’s bank account. If the participant opened a special bank 
account to store money from the program, the deposits might have been salient because they 
were clearly marked, or less so if the participant often forgot to check the account or found it 
difficult to access the money. Lastly, Family Rewards distributed the incentives to parents and 
high school students, unlike the EITC, which is paid to the head of household. 

The following section summarizes earnings in each of the domains, focusing on the pat- 
terns in Years 2 and 3 since these were the only years when all enrolled families were eligible 
for rewards for a full 12 months. 

Education 

• Almost all families (97 percent) earned at least one education reward due 
mainly to the attendance and report card rewards. In Year 2, for example, of 
families with students who earned an education reward, 88 percent earned at 
least one for attendance and 80 percent for grades. Students continued to 


6 lnternal Revenue Service (2016). 

7 Kneebone and Holmes (2015). 
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Table 2.1 


Rewards Earned by Families in Years 1 Through 4 


Outcome 

Year l a 

Year 2 

Year 3 

Year 4 b 

Years 1-4 

Family earned at least 1 reward (%) 

95.0 

96.3 

94.4 

11 A 

98.6 

Among families who earned at least 1 reward 

Average number of rewards earned 

23.1 

36.9 

35.0 

10.5 

100.1 

Average reward amount earned c ($) 

1,598 

2,314 

2,039 

624 

6,241 

Education rewards 

Family earned at least 1 education reward (%) 

91.5 

93.3 

88.7 

57.9 

97.3 

Attendance 

82.5 

88.4 

86.9 

54.0 

96.0 

State core exam 

57.3 

66.5 

36.3 

0.0 

82.1 

SAT/ACT 

1.0 

6.6 

8.9 

2.8 

19.0 

Report card 

77.7 

80.2 

73.7 

38.9 

89.9 

Among families who earned at least 1 education reward 

Average number of education rewards earned 

16.5 

28.5 

27.4 

9.4 

73.4 

Average education reward amount earned ($) 

777 

1,224 

974 

256 

2,943 

Health rewards 

Family earned at least 1 health reward (%) 

81.0 

78.8 

76.8 

46.6 

89.8 

Annual physical 

70.1 

70.4 

68.9 

32.5 

86.0 

Semiannual dental 

71.6 

70.7 

68.1 

34.9 

84.7 

Among families who earned at least 1 health reward 

Average number of health rewards earned 

5.7 

6.6 

6.3 

3.3 

18.0 

Average health reward amount earned ($) 

570 

663 

626 

334 

1,805 

Work rewards 

Family earned at least 1 work reward (%) 

39.6 

46.3 

46.1 

37.6 

56.9 

Full-time employment (%) 

39.5 

45.9 

46.0 

37.3 

56.2 

High school equivalency certificate (%) 

0.4 

0.8 

0.5 

0.2 

2.0 

Among families who earned at least 1 work reward 

Average number of work rewards earned 

5.8 

8.1 

8.4 

3.1 

19.5 

Average work reward amount earned ($) 

872 

1,216 

1,259 

475 

2,926 

Sample size 

1,230 






(continued) 
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Table 2.1 (continued) 


SOURCE: MDRC calculations using Children's Aid Society's Family Rewards program data. 

NOTES: Sample sizes may vary because of missing values. 

Year 1 covers reward activities that occurred between September 2011 and August 2012. Year 2 
covers reward activities that occurred between September 2012 and August 2013. Year 3 covers reward 
activities that occurred between September 2013 and August 2014. Year 4 covers reward activities that 
occurred between September 2014 and December 2014. 

Italics indicate outcomes calculated for a subset of the full sample. 

a Random assignment occurred between September 2011 and February 2012 and prevented many 
participants from receiving a full year of rewards. 

b Y ear 4 only permitted participants to earn during four months of the program year. 
c The lowest and highest amounts earned in Year 1 were $20 and $8,851. The lowest and highest 
amounts earned in Year 2 were $38 and $8,200. The lowest and highest amounts earned in Year 3 were 
$40 and $7,550. The lowest and highest amounts earned in Year 4 were $10 and $3,110. 


qualify for the attendance reward at a high level in Year 3 but the percentage 
of families with students who claimed the report card reward declined to 74 
percent. This reward turned out to be more difficult to administer than ex- 
pected — the two cities had different grading scales, some courses were inel- 
igible, report cards had to be distinguished from “progress reports,” and each 
school distributed different numbers of report cards. CAS-Central issued pol- 
icy changes to address these challenges, but they may have led to some con- 
fusion or disappointment for families as they attempted to claim the reward. 
Further, the report card reward was meant to facilitate ongoing interaction 
between advisors and students as students brought in each newly issued re- 
port card to the NPO, but this interaction did not occur in most NPOs. 

• More families earned the reward for passing at least one state exam in Year 2 
(67 percent) than in Year 3 (36 percent). 8 Almost half of the students entered 
the program in the tenth grade, and very few students took these exams dur- 
ing their senior year (Year 3 for entering tenth-graders). 

• Few families (19 percent) ever earned the incentive for taking the SAT or 
ACT college entrance exams. Most of the students who earned it were in the 
Bronx. In Memphis, schools offered the ACT for free but students had to 
earn a certain score to qualify for the reward. 


8 These figures are reported among students who earned any education reward. Ninety-three percent earned 
an education reward in Year 2 and 89 percent in Year 3. 
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• In general, families in Memphis earned less in education rewards than 
families in the Bronx. As a group, students in Memphis families had lower 
grades (and the amount of the report card incentive was scaled to the 
grade), passed fewer state exams, and did not claim the college entrance 
exam reward as often. 9 

Health 

• Families were highly engaged in the health domain, which offered rewards to 
the entire family including all children 19 years of age and under. Ninety 
percent of families earned at least one health reward, and very similar pro- 
portions of families claimed the reward in Years 1,2, and 3. Because Year 4 
was just four months in length, and insurance providers only pay for exami- 
nations at a certain periodicity, many participants could not claim this reward 
during this time. 

• Memphis adults faced barriers to claiming the dental reward because these 
visits were not covered by the public health insurance program in Tennes- 
see as they were in New York State. In Year 2 (data not shown), three- 
quarters of Memphis adults did not earn any rewards for dental visits. The 
family advisors made a concerted effort to build relationships with provid- 
ers who offered free or low-cost cleanings, which increased access to this 
reward in later years. 

Work 

• Fifty-seven percent of families earned at least one work reward, and almost 
all of these rewards were for full-time employment. The proportion of fami- 
lies that claimed the full-time work reward was the same in Years 2 and 3 at 
around 46 percent. Of the families that earned it, they did so consistently, 
claiming the reward an average of 8 out of 12 months. 

• Despite the fact that 48 percent of adults in the Bronx and 31 percent in 
Memphis did not have a high school diploma or equivalency certificate at the 
start of the study, almost no adults claimed the reward for earning the Gen- 
eral Education Development (GED) certificate. The GED test was over- 
hauled while the program was in operation. The advisors tried to use this 

9 It should be noted that the grading scales in the two cities were different. In Memphis, the range of scores 
that constituted a C was larger than in New York City, while the range of scores that constituted a B or an A 
was smaller. (See Table 1.1.) 
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overhaul to motivate adults who were close to passing it to buckle down and 
get it done, but this campaign was not successful. 

The pattern of reward receipt shows that the earlier Family Rewards 2.0 implementation 
report did not accurately predict the future course of reward earnings in the program. Despite 
improvements to the quality of advising and other changes to outreach strategies and manage- 
ment techniques that will be reviewed in the next section, reward earnings did not increase. In 
particular, advisors were not able to increase the number of rewards participants received for 
full-time employment by helping more participants find employment or increase their work 
hours. By Year 2, a larger share of households were claiming this work reward than those in 
Family Rewards 1.0, but the proportion did not increase in Year 3. The next section explores 
how the program set out to engage families in Years 3 and 4 of the program. 


The Advisor-Participant Relationship at the End of the Program 

Each NPO hired three advisors to work directly with families. Most were women who shared 
characteristics with the majority of families in their caseloads in tenns of race, ethnicity, and 
spoken language. Most were generalists coming from the social services or nonprofit sectors, 
but two were trained social workers and one was a fonner teacher. Each NPO had a program 
manager who oversaw the advisor’s work on site, and CAS-Central hired a social worker and 
business professional to provide additional technical assistance. 

Each advisor served about 100 families (with at least one parent and one high school 
student per family), though some worked with fewer families because they were part time or 
less skilled, and some had many more for periods of time because of turnover in the position. 
Advisors had three main responsibilities: 

1. Outreach: advertising the incentives, planning events, and maintaining a high level 
of general interest in the program. 

2. Customer Setyice: helping participants fill out coupons and understand program 
rules, and serving as an on-site liaison between staff who reviewed documentation 
at the payment processing center and participants. A customized salesforce applica- 
tion that included screens with payment decisions and notes from guidance sessions 
and outreach attempts facilitated this relationship. 

3. Guidance: building personal relationships with the adults and high school students 
to provide a mix of emotional and instrumental support in reaching goals. 

The original design model called for a triage approach, whereby advisors were to en- 
gage in these modes of interaction with all participants but offer more assistance to those who 
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were struggling to earn rewards. Each adult and high school student was expected to receive 
guidance sessions twice a year to update an action plan called a Family Earning Plan and, at the 
end of the year, review earnings from the past period. Advisors were to contact participants 
whom CAS-Central had identified as low reward-earners usually for guidance sessions and 
other supports. 

As noted in the earlier implementation report, the nature of the advisors’ work evolved 
significantly in the first two years of the program. 10 Advisors focused mainly on building 
relationships in the first year, which involved advisors and clients getting to know each other 
and establishing trust and familiarity. One-on-one sessions at this time were largely transaction- 
al or customer service oriented. Advisors had few referral partners and did not know how to 
address some of the major life issues that arose, such as housing dislocation, serious health 
problems, domestic violence, infidelity, divorce, substance abuse, teenage pregnancy, and 
incarceration. 

Recognizing that advisors needed more training to go beyond customer service and en- 
courage behavioral change, CAS-Central introduced motivational interviewing (MI) in the first 
half of Year 2. MI is a widely used, directive approach to counseling in which the counselor 
uses a set of techniques to explore an individual’s ambivalence about change. 11 Evaluations 
have found that MI can be an effective way of encouraging engagement with a program or 
improving the primary outcomes of treatment. 12 It took several months of in-service support for 
all advisors to develop a minimum level of proficiency in the techniques. Observations of 
guidance sessions that took place during the second half of Year 2 showed that the content and 
feel of the sessions had in fact improved. Participants talked more about their goals, and took 
the lead in creating plans for overcoming barriers to earning rewards. 

This change was positive, but participants did not receive as much of this quality ad- 
visement as intended. Table 2.2 summarizes the proportion of adult and high school student 
participants who received guidance as well as the proportion of them who received different 
types of guidance in the Bronx and Memphis in Years 2 and 3. In Year 2, most adults had one 
guidance session in the Bronx and Memphis (86 and 65 percent, respectively), but only a 
minority of these individuals had two or more such sessions (34 and 25 percent, respectively). 
Smaller percentages received referrals (13 and 19 percent, respectively) or discussed barriers or 
next steps (33 and 25 percent, respectively) with their advisors. These last outcomes were 


10 Dechausay, Miller, and Quiroz-Beccera (2014). 
"Miller and Rollnick (2002). 

12 Hettema, Steele, and Miller (2005). 
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Table 2.2 


Advising Received by Participants in Years 2 and 3, by City 


Outcome 

Bronx 

Year 2 

Year 3 

Memphis 

Year 2 Year 3 

Adult participants (%) 

Participant had a guidance session 1 

87.3 

90.8 

72.4 

76.4 

Had at least 1 FEP review 

85.7 

88.7 

64.8 

66.4 

Had 2 or more FEP reviews 

33.6 

64.4 

25.0 

26.9 

Ever received a referral 

13.1 

62.8 

18.5 

38.8 

Ever discussed barriers to earning rewards or next steps 

32.7 

29.0 

24.7 

12.8 

Participant received customer service 15 

69.5 

73.0 

72.6 

43.0 

Participant received outreach' 

87.9 

81.2 

93.7 

68.8 

Sample size 

655 

655 

631 

631 

Student participants ' 1 (%) 

Participant had a guidance session 3 6 

86.6 

88.6 

83.4 

82.8 

Had at least 1 academic/FEP review 

86.6 

88.0 

81.7 

80.4 

Had 2 or more academic/FEP reviews 

67.7 

64.8 

20.2 

25.6 

Ever received a referral 

2.4 

21.2 

3.5 

4.1 

Ever discussed barriers to earning rewards or next steps 

26.4 

34.0 

12.4 

8.0 

Participant received customer service b f 

69.4 

75.2 

63.8 

39.7 

Participant received outreach' 

67.1 

64.8 

23.6 

13.1 

Sample size 

656 

656 

687 

687 


(continued) 


indicators of the intensity of the guidance, or the likelihood that the guidance would generate 
behavioral change in people who were not at the time on the path to earning certain rewards. 
Most adult participants received customer service and general outreach. 

This trend in Year 2 was similar for high school participants in Memphis, with 82 per- 
cent of students having had one guidance session, but only 20 percent having had two or more. 
The percentage of these students that received referrals or discussed barriers and next steps in 
guidance session was very small (4 and 12 percent, respectively). In contrast, in the Bronx, 
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Table 2.2 (continued) 


SOURCE: MDRC calculations using the Children's Aid Society's Family Rewards program data. 

NOTES: FEP = Family Earning Plan. 

Sample sizes may vary because of missing values. 

Year 2 covers activities that occurred between September 2012 and August 2013. Year 3 covers 
activities that occurred between September 2013 and August 2014. 

“Guidance includes creating FEP, reviewing FEP, discussing next steps, discussing barriers to 
earning rewards, discussing public benefits, and making a referral, discussing earnings goals, receiving 
support from the Family Resource Fund, or discussing family finances. 

b Customer service includes group check-ins, group workshops, discussing referral outcomes, 
submitting bank account information, adjusting family composition or eligibility, advisor signature for 
high school equivalency certificate completion, answering earnings questions, help-line calls, and coupon 
submission support. 

“Outreach includes program campaigning (outreach to particular participants with information or 
reminders), scheduling coupon book pick-up, program reengagement calls, sending e-mails, sending 
letters, and conducting automated calls. 

d This category includes students enrolled in ninth or tenth grade at baseline. 

“This category also includes report card review for students. 

' I ll is category also includes advisor signature for student report cards. 


advisors and high school participants interacted much more. Eighty-seven percent of these 
students received one guidance session, while 68 percent received two or more. These sessions, 
however, clearly did not focus entirely on motivating behavioral change since only 3 percent of 
the students received a referral and 26 percent discussed barriers. That said, a stark difference 
began to emerge in Year 2 between the guidance students received in Memphis and the Bronx. 
The advisors in the Bronx found that the students’ ability to travel to the NPO on their own 
facilitated these interactions, and though not technically required by the program, they asked 
students to come in every time they received a report card to ensure a high level of contact. 13 


13 ln the first year of the program, students were required to bring in the coupon for signature in order to 
receive payment. Program designers intended for this meeting to occur quarterly and for advisors to use the 
students’ report cards to structure a conversation about their academic performance, and remind them about the 
potential reward earnings. This process did not work as well as hoped in practice because (a) many students in 
the Bronx received more than four report cards leading to a large number of meetings; (b) attending the in- 
person meetings was difficult for many families in Memphis who relied on cars, and so it was typical for 
parents in Memphis to bring several report cards at one time on behalf of their children; and (c) the meetings 
were largely transactional, appointments at which the advisor simply signed report cards and assisted with 
other administrative issues. In Year 2, the program designers removed the report card signature requirement 
because of the problems noted in the first year of the program and replaced it with a single mandatory meeting 
between students and advisors at the beginning of the school year. This meeting, called an “academic review,” 
provided an opportunity for advisors to discuss students’ performance in school and plan for the year. Students 
who struggled to earn rewards after that point would be identified in the low-eamers reports, which would 
prompt advisors to reach out to these individuals to offer additional support. Advisors in Memphis reported a 

(continued) 
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Overall, most participants in Year 2 received only one intensive guidance session in 
which the advisor might have used MI techniques to promote engagement, although a lot of 
outreach and customer service occurred in this period (similar to what participants in Family 
Rewards 1.0 received). 

In Year 3, there was some improvement in indicators of the intensity of the relationship 
between advisors and participants in the Bronx. More adults in the Bronx received two or more 
guidance sessions (65 percent in Year 3, compared with 34 percent in Year 2), and many more 
adults received referrals (63 percent in Year 3, compared with 13 percent in Year 2). Students in 
the Bronx who received at least one guidance session received more referrals in Year 3 (21 
percent of students in Year 3, compared with 3 percent in Year 2). This improvement was the 
result of a focused effort to develop referral partnerships in New York City, especially those 
with workforce development providers. 

In contrast, advisors in Memphis did not increase contact with participating adults or 
students in Year 3. Advisors in Memphis reported that because the guidance sessions were not 
mandatory and participants had to arrange car rides to get to the NPOs, many stopped visiting 
the NPOs after the first year. As a result, advisors progressively shifted to delivering the 
majority of their guidance sessions over the phone, which they acknowledged was far from 
ideal. Advisors in Memphis did increase their use of referrals (39 percent of adults in Year 3, 
compared with 19 percent in Year 2), but these new referrals were primarily for dental services 
for adults. Advisors described a lack of quality workforce development providers. Advisors also 
reported lower rates of outreach and customer service in Year 3. This decrease may have been 
the result of a new performance management approach introduced in Year 3 that resulted in 
advisors in Memphis conducting more home visits. 

In an effort to further refine the guidance component, the program designers established 
quarterly benchmarks that provided specific targets for advisors to meet within the education, 
health, and work domains. Advisors were offered small incentives to motivate them to achieve 
these benchmarks. Each quarter, the program designers and CAS-Central defined a target group 
of adults and high school students based, for the most part, on their reward earnings outcomes, 
as well as provided NPOs with weekly reports on each advisor’s progress toward reaching the 
benchmarks. For example, from March to May 2014, the benchmarks directed advisors to 
increase the number of adults earning work rewards by five each, achieve a 30 percent increase 
in the number of adults earning a medical and dental health reward, and ensure that 75 percent 
of their student caseload had earned at least $75 in grades rewards in Year 3. Advisors in 
Memphis attempted to meet the benchmarks by more aggressively reaching out to eligible 


steep decline in visits to the NPO by participants after this policy change. Neither NPO in the Bronx announced 
this change to maintain a high level of contact with students. 
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participants, and, in most cases, offering to personally pick up paperwork participants had not 
turned in or conduct guidance sessions in participants’ homes. The time that advisors spent 
traveling to home visits affected their ability to provide services to participants generally, but it 
was seen as the only way to achieve the benchmarks. In most quarters, the benchmarks were not 
achieved, but the advisors reported that they appreciated them because they brought greater 
focus to their work. 

In the end, the guidance component involved a lot of operational challenges related to 
training advisors, making contact with participants, and targeting the right individuals at the 
right times. Student participants in the Bronx had more ongoing contact with advisors than 
student participants in Memphis, who rarely met with their advisors one-on-one. While families 
in the Bronx earned more in education rewards than did families in Memphis, it is impossible to 
conclude that this difference was due to the relationship with their advisors because of many 
confounding factors. At best, it seems that the intensity of the guidance that participants in the 
Bronx received contributed to their engagement in the program but was not transformative. 
Even after advisors began using MI, participants did not experience a significant, positive 
change in the trajectory of their reward earnings. Dosage may have played a role in this result 
— participants faced major life challenges and had relatively little contact with their advisors 
even in the best cases. 


Program Costs 

Over about three years, the program spent $13,459 on the typical participating family (Table 
2.3). 14 The largest spending category was rewards paid to families (48.3 percent). It included the 
cash rewards in all three of the domains (education, health, and work), expenses associated with 
an enrollment incentive paid to families when they signed up for the study, and a discretionary 
pot of money called the Family Resource Fund that was available to advisors. 

The remaining program costs (51.7 percent) were associated with the administration of 
the program by CAS-Central and the NPOs, including staff salaries, fringe benefits, consultants, 
the administration of payments and support services for participating families, and other costs. 
Staff salaries at CAS-Central accounted for the largest administrative cost ($2,452 per family). 
The costs associated with the NPOs in New York and Tennessee were the second largest 


14 The direct cost estimate is based on program expenditure reports from 2011 to 2014. All dollar amounts 
have been converted to 2015 dollars using the Consumer Price Index for all urban consumers. As a result of 
this approach, the cost of “reward payments to participating families” ($6,507) in Table 2.3 is slightly different 
from the “average reward amount earned” per family ($6,241) presented in Table 2.1, which was based on 
individual-level MIS data and was not adjusted for inflation. 
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Table 2.3 


Cost per Family 
(in 2015 Dollars) 


Cost Category 

Cost ($) 

Percentage of 
Total (%) 

Reward payments to participating families 



Education rewards 

3,030 

22.5 

Health rewards 

1,713 

12.7 

Work rewards 

1,764 

13.1 

Subtotal 

6,507 

48.3 

Non-reward costs 



Central operations 



Staff salaries 

2,452 

18.2 

Fringe benefits 

617 

4.6 

Consultants 

396 

2.9 

Other costs 

1,200 

8.9 

Local operations 



Partner organizations 

2,287 

17.0 

Subtotal 

6,952 

51.7 

Total cost 

13,459 

100.0 

Non-reward cost per dollar of reward payment 

1.07 


Sample size 


1,230 


(continued) 


administrative expense ($2,287). Fringe benefits for staff cost $617 per family served. Consult- 
ants cost $396 per family, or 3 percent of total costs. The notes to Table 2.3 include additional 
information on each spending category. Finally, the bottom row of the table indicates that for 
every dollar transferred to families for rewards, $1.07 was spent on non-reward costs. 

To put the costs of the cash transfer in perspective, it is important to remember that 
Family Rewards was a small and new program. Size and newness increase the amount of non- 
reward costs, especially the administrative cost of processing payments; more established and 
larger programs typically serve participants at a lower cost. For example, Temporary Assistance 
for Needy Families (TANF) spends approximately 17 cents per dollar of payment, Supple- 
mental Nutrition Assistance Program spends 20 cents per dollar of payment, and Medicaid pays 
5 cents per dollar of payment, although these programs also do not offer guidance to help 
families earn payments. 15 


l5 The research team obtained data on the TANF costs and payments from the financial data that states 
submit to the Administration for Children and Families (data reporting form ACF-196). It obtained Food 
Stamp Program outlays and obligations data from the Food and Nutrition Service in the U.S. Department of 

(continued) 
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Table 2.3 (continued) 


SOURCES: Program expenditure reports from 2011 to 2015, MDRC calculations from management 
information system data, and phone interviews with program staff. 

NOTES: Estimates have been adjusted for inflation and reflect discounting. 

In each domain the cash transfer value from reward payments is presented as a cost. In Tables 3 
and 4, an amount equal to the reward payments is presented as a benefit to participants. These 
amounts cancel each other and are cost and benefit neutral from the perspective of society. 

Description of cost categories: 

Reward payments to participating families include the conditional cash transfer payments (education, 
health, and work), the Family Resource Fund (a discretionary fund intended to assist each family earn 
certain rewards), and small incentive payments for opening a savings account. Resource fund and 
enrollment incentive payments were not tracked separately and therefore cannot be separated from 
conditional cash transfer payments. 

Non-reward costs include all resources required to operate the program beyond the reward payments 
to participating families, such as the services to inform participants of available rewards, support 
participants in earning rewards, and process reward payments. 

Staff salaries are remuneration paid to senior managerial staff (presidents, directors, and senior 
managers), programmatic staff (program associates, coordinators, assistants, advisors, and reward 
specialists), and administrative staff (financial and business analysts, general counsels, and 
information technology and database staff). Staff were responsible for the development of a payment 
tracking system, processing administrative records to determine if automatically verified rewards had 
been earned, creating coupon books for reward payments that required families to submit documents 
showing that they earned the reward, verifying requirements for coupon payment rewards were met, 
maintaining up-to-date bank account information to make sure payments were disbursed to the correct 
accounts, issuing “earnings statements” each payment period to mail to families, creating and 
maintaining a helpline to answer questions, making payments to families who earned rewards, 
marketing the program, general program management, and oversight of nonprofit organizations. 
Fringe benefits - Fringe benefits for central operations staff in New York and Tennessee. 

Consultants - Additional administrative support for Children's Aid Society to help the sites with 
managing grants, advertising, supporting audits, and general temporary staffing. 

Other costs - Costs other than expenditures for personnel services. These expenditures include the 
cost of travel associated with program staff meeting with participants near their places of residence. 
Other costs also include supplies, outreach materials, insurance, rent for office space, and other 
indirect costs. 

Partner organizations - All costs associated with the four partner organizations responsible for 
implementing Family Rewards, including Porter-Leath and Memphis FIOPE in Tennessee and 
Children’s Aid Society and BronxWorks in New York. These costs include program orientations, 
refresher sessions, coupon book distribution, customer service, social events, and workshops. 


Conclusion 

Because few CCT programs include a guidance component, there were no direct comparisons 
for the program that CAS-Central and its partners delivered. Family Rewards 2.0 aimed to bring 
about behavioral changes in participants by rewarding them not only for behaviors in which 


Agriculture. It obtained financial data on the Medicaid program from the Center for Medicare and Medicaid 
Quarterly Expense Report. 
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they regularly or periodically engaged before entering the program, but for engaging in new or 
sustained behaviors that would increase their human capital. Program developers reduced the 
number of the program’s rewards relative to Family Rewards 1.0 because of initial budget 
constraints, but they expected that the investment in advisors would be a positive addition by 
helping participants create action plans to maximize their receipt of the available cash rewards 
and overcome barriers along the way. 

CAS-Central delivered most of the incentives for Family Rewards 1 .0 and made small 
changes to the verification rules for the new program in an effort to make the incentives easier 
to claim. However, the new rewards for report cards, designed to reward participants’ effort and 
educational input rather than their performance outcomes, raised difficult practical issues and 
did not generate the kind of ongoing interaction between students and advisors that the program 
designers had expected. 

The guidance component was completely new and evolved substantially over the 
course of the program. Transportation obstacles in Memphis impeded contact between advisors 
and participants, and thus the component never gained traction. Advisors in the Bronx had 
greater contact with participants, but the frequency of the interaction was still less than antici- 
pated (except, perhaps, for student participants). 

Most families earned rewards for education and health during the first three years, and 
the share of families enrolled in Family Rewards 2.0 that claimed the reward for full-time 
employment was greater than that of families enrolled in Family Rewards 1.0. Still, the percent- 
age of families earning work rewards did not increase over time. This result was disappointing 
considering the program’s many components aimed at changing the underlying behaviors in 
participants that would increase reward receipt, including intensive training for advisors, new 
targeting approaches, and the development of referral networks. 
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Chapter 3 

Impacts on Income and Material Well-Being 


In the short run, the goal of Family Rewards was to reduce current poverty and material 
hardship through its direct cash reward payments — or conditional cash transfers (CCTs) — 
and through any immediate increases it might generate in parents’ earnings from employment. 
The previous chapter showed that families enrolled in Family Rewards 2.0 earned significant 
cash transfers during the program years, just over $2,000 per year. 

This chapter examines the effects of the cash transfers on a broad range of income and 
well-being outcomes. Focusing on Year 2, roughly midway through the program, it considers 
how, and to what extent, participation in Family Rewards 2.0 helped to alleviate poverty and 
hardship. In contrast to Family Rewards 1.0, Family Rewards 2.0 targeted families who were 
receiving Temporary Assistance for Needy Families (TANF) or Supplemental Nutrition 
Assistance Program (SNAP, or “food stamps”), in order to direct resources to the neediest 
families, and offered fewer rewards in each of the three domains. As a result, a central question 
is whether the program led to comparable or even larger impacts on income and well-being for 
these families. The next several sections examine impacts on a range of measures of income and 
well-being. The chapter closes with a brief comparison of the findings from Family Rewards 
1.0 and 2.0. 


Summary of Impacts on Income and Well-Being 

Overall, the results show that Family Rewards 2.0 achieved some of its immediate goals. 
Roughly two years after families enrolled in the program, positive effects on various dimensions 
of household income and well-being were evident. Family Rewards 2.0 increased average 
household income, reduced the proportion of families living in poverty, and increased the 
portion of families with savings. It also kept families in the program group connected to 
mainstream banking institutions. In addition, program group members were more likely to 
report higher levels of financial well-being, life satisfaction, and happiness, compared with their 
control group counterparts. 

The general pattern of findings in this report are consistent with early results from the 
Family Rewards 1.0 evaluation, although the earlier program produced larger effects on income 
and poverty and financial well-being around 1 8 months of follow-up (when families were still 
eligible to earn rewards). However, there was no evidence that families in Family Rewards 2.0 
experienced less housing-related material hardship in the initial years, an effect observed in 
Family Rewards 1.0. One likely reason for this difference is that Family Rewards 2.0 offered 
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fewer rewards and transferred less money to families. In addition, about half of the rewards that 
families earned were earned by — and paid to — high school students, funds that may not have 
contributed as directly to meeting basic family needs. 


Data and Methods 

The analyses presented here draw on the 24-month survey, as well as administrative records 
data on TANF and SNAP receipt provided by the local administrative agencies. The survey was 
administered to the full study sample and provides extensive data on 2,016 families — or 82 
percent of the full sample. 1 The survey interviews were conducted between October 2013 and 
May 2014, the period roughly corresponding to Year 2. Thus, given the timing of the survey, 
the rewards earned in Years 1 and 2 (see Chapter 2) are most relevant for understanding the 
program’s effects on income and well-being while families were in the program. The chapter 
also draws on qualitative data obtained through in-depth interviews with a small sample of 
program participants. 

Box 3.1 explains how to interpret the tables that show estimated program impacts that 
are presented throughout the remainder of this report. These tables cover a large number of 
impact estimates that are relevant to family poverty, hardship, and economic security and in the 
three domains in which the program provided rewards: children’s education, family preventive 
health care, and parents’ work and training. The research team calculated the estimates of 
program impacts controlling for a range of pre-random assignment background characteristics, 
such as the parents’ race or ethnicity, education level, marital status, and employment status. 
Appendix E provides more infonnation on the data and analysis methods and is available on the 
MDRC website at www.mdrc.org. 

As the number of outcomes that are examined increases, the probability of obtaining 
impacts that are statistically significant simply by chance also increases. In order to address this 
issue, the impact analysis presented in each of the chapters distinguishes between a limited 
number of primary outcomes in a given domain versus secondary outcomes. If the research 
team found that the program affected primary outcomes in a given domain (for example, 
income and poverty), then it assigned more weight to the effects found for secondary outcomes 
within that domain (material hardship, for example). In contrast, if there were no effects on a 


Yhe survey provides information about sample members on a broad set of topics, including participation 
in employment and education activities, health care status, employment and job characteristics, household 
composition, and child outcomes. Appendix F provides an analysis of survey response and is available on the 
MDRC website at www.mdrc.org. 
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Box 3.1 

How to Read Impact Tables in This Report 


In the context of this evaluation, an “impact” is a measure of how much Family Rewards 2.0 
changed outcomes for program participants. All the tables in this report that show impacts use 
a similar format, illustrated in the table exceipt below, which presents data on a material 
hardship and a well-being outcome that were obtained from parents’ reports on the 24-month 
survey. The top row, for example, shows that 41.4 percent of respondents in the program 
group had not paid their rent in full or made a full mortgage payment in the 12 months before 
the time of the survey interview, compared with 42 percent of control group respondents. 

Because families were assigned randomly either to a program group or to a control group, the 
effects of the program can be estimated by the difference in outcomes between the two groups. 
The “Difference” column in the table shows the differences between the two research groups’ 
outcomes — that is, the program’s estimated impacts on the outcomes. For example, the 
estimated program impact on paying full rent or mortgage can be calculated by subtracting 
41.4 percent from 42.0 percent, yielding a reduction, or estimated impact, of -0.6 percentage 
points. 

The p-value shows the probability that this difference, or impact, arose by chance. In the table 
below, the difference between the program and control groups in paying the full rent or 
mortgage has a 79 percent probability of arising as a result of chance rather than as a result of 
the Family Rewards program. In contrast, the difference on the measure of financial well-being 
(bottom row) has a 2 percent probability of having arisen by chance. For this evaluation, only 
differences that have a 10 percent probability or less of arising by chance are considered 
“statistically significant” and therefore represent true program effects. The number of asterisks 
indicates whether the impact is statistically significant at the 1 percent (***), 5 percent (**), or 
10 percent (*) level. 

The final column shows the effect size, which is presented for selected impacts. The effect size 
is the difference between the program and control group outcomes divided by the “standard 
deviation” of the outcome (a measure of its variability). Expressing an impact in standard 
deviation units helps to interpret its size, particularly when the outcome is in nonstandard units, 
such as scores on a scale, as in the second row. In contrast, effect sizes are not shown for 
impacts that are easier to interpret, such as impacts on dollar amounts or percentages. 


Impacts on Two Material Hardship and Well-being Outcomes, 
from the 24-Month Survey 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Effect 

Size 

Did not pay full rent or 
mortgage (%) 

41.4 

42.0 

-0.6 

0.791 


Financial well-being 
(4 = low; 16 = high) 

9.3 

9.1 

0.2** 

0.021 

0.103 
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primary outcome within a domain, the research team gave the effects on secondary outcomes 
less weight in the analysis. 2 Although this strategy helped to avoid placing undue weight on 
false positives, caution should be used when interpreting impacts that do not appear to be part of 
a larger pattern of impacts within a given set of measures. In addition, and as noted in Chapter 
1 , the main analysis focuses on the Bronx and Memphis sites combined (or a pooled analysis), 
and Chapter 7 looks at how the effects vary by city and across other subgroups. 


Impacts on Income and Poverty 

To estimate household income and poverty, families completing the 24-month survey were 
asked to list their sources of income and total household income for the calendar month before 
the month of their survey interview. 3 As the top panel of Table 3.1 shows, the estimated average 
monthly household income in that month was $1,636 for the program group (including rewards 
payments) and $1,498 for the control group, representing a statistically significant gain of $138 
— or about a 9 percent increase relative to the control group members’ income. On average, 
families in the program group earned an additional $184 each month in reward payments in the 
year before the interview. If reward payments are excluded, the average monthly household 
income at the time of the interview was similar for the two study groups. 

The next several rows present the distribution of income relative to the federal poverty 
line. 4 The table shows the distribution of this poverty measure across four levels, including 
“severe poverty,” which is defined as income below 50 percent of the poverty threshold. The 
estimate, which includes reward payments in the calculation of average household income, 
indicates that Family Rewards 2.0 altered the distribution of families across the different levels 
of poverty, in particular reducing the proportion of households in severe poverty and increasing 
the proportion with income at or above 130 percent of the federal poverty threshold. While 33.7 


2 As a sensitivity test, standard errors for impact estimates for the primary outcomes were adjusted for mul- 
tiple hypothesis testing. Appendix E (available on the MDRC website at www.mdrc.org) presents the results, 
which are generally similar to those presented in the main report. 

^Respondents were instructed to exclude tax refunds and program reward payments, and to include in- 
come from all other sources such as their job(s), jobs of other household members, food stamps, child support, 
TANF, Supplemental Security Income, and unemployment insurance, among other sources, for everyone 
living together in the household. They were also asked to exclude EITC payments because of the difficulty of 
converting them into a monthly amount. The research team combined the reported sources of income and used 
them to calculate household income and poverty relative to the federal poverty threshold, using the 2013 or 
2014 poverty guidelines, depending on when a respondent was interviewed. Given the particular definition, 
readers should be cautious comparing the poverty estimates presented here with estimates from other published 
sources. 

4 The research team estimated annual household income based on the one-month household income snap- 
shot gathered at the time of the survey interview. 
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Table 3.1 


Impacts on Income and Income Sources 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Income and poverty 

Average total household income in month prior to 

interview (excluding Family Rewards payments) 2 ($) 

1,452 

1,497 

-45 

0.332 

Average monthly Family Rewards payments 

received during Year 2 ($) 

184 




Average total household income in month prior to 

interview (including Family Rewards payments) 2 ($) 

1,636 

1,498 

138 *** 

0.004 

Household income at or below the federal poverty 

level (including rewards) 2 ’ 15 (%) 

73.5 

78.3 

_4 8 ** 

0.012 

Total household income in prior year as a percentage 
of the federal poverty level (including rewards) 2,15 (%) 

Less than 50% 

26.6 

33.7 

_7 j *** 

0.001 

50% to 100% 

46.9 

44.6 

2.3 

0.313 

101% to 129% 

14.1 

13.2 

1.0 

0.539 

130% or more 

12.3 

8.5 

2 g *** 

0.006 

Income sources (%) 

Household income source in month prior to interview 

Respondent's earnings 

60.0 

60.5 

-0.5 

0.794 

Other household members' earnings 

23.6 

23.6 

0.0 

0.996 

Food stamps 

82.5 

81.7 

0.8 

0.632 

Child support 

25.0 

22.7 

2.3 

0.210 

Temporary Assistance for Needy Families (TANF) 

or other cash assistance 

7.4 

8.2 

-0.8 

0.515 

Unemployment insurance 

4.7 

5.7 

-1.0 

0.311 

Special Supplemental Nutrition Program for Women, 

Infants, and Children (WIC) 

11.5 

10.7 

0.9 

0.538 

Heating or cooling assistance 

7.4 

5.8 

1.6 

0.153 

Free or reduced-price school lunch 

75.8 

71.6 

4.2 ** 

0.033 

Supplemental Security Income or Social Security 

Disability Insurance 

32.4 

29.9 

2.5 

0.199 

Other 

4.6 

4.2 

0.4 

0.655 

Sample size (total = 2,016) 

1,025 

991 




(continued) 
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Table 3.1 (continued) 

SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Italics indicate outcomes calculated for a subset of the full sample. 

a Monthly household income amounts equal to or greater than $10,000 were excluded from this 
calculation. About 4.9 percent of the sample was excluded from the income measures because 
respondents did not know or refused to provide the information. An additional 0.2 percent of the sample 
was excluded because the income provided was over $10,000. 

b Annual household income was calculated by multiplying the respondent's income in the month 
prior to the survey interview by 12. For program group members, it includes Family Rewards payments 
earned during program Years 2 and 3. The federal poverty level was calculated based on annual income 
(monthly income multiplied by 12) and the household size at the time of the survey. The poverty 
threshold was measured according to the 2013 or 2014 Poverty Guidelines, depending on when a 
respondent was interviewed. 


percent of control group members’ income fell in the severe poverty category, only 26.6 percent 
of the program group members’ income fell in that category, representing a statistically signifi- 
cant drop of 7.1 percentage points. Similarly, 8.5 percent of the control group members earned 
an income at or above 130 percent of the federal poverty threshold compared with 12.3 percent 
of the program group, a statistically significant gain of 3.8 percentage points. This type of effect 
on poverty reduction is generally expected in programs that offer cash transfers and was 
expected in Family Rewards 2.0. Despite being offering fewer opportunities to earn rewards 
than participants in the original program, the majority of families in Family Rewards 2.0 earned 
cash transfers, and the total annual payments per family averaged around $2,300 during the 
second year of the program. Overall, the effect on poverty reduction observed in Family 
Rewards 2.0 is consistent with, although smaller than, the effect seen in Family Rewards l.O. 5 

The 24-month survey also asked respondents whether they had received income or ben- 
efits in the prior month from a variety of sources, including job earnings, government benefits, 
and child support. Program and control group members reported similar income sources and 
there was no evidence that the program increased or decreased the likelihood of receiving 
income from particular sources. 

In addition to self-reported income sources, the research team examined administrative 
records data to estimate the program’s effects on benefits that participants received from TANF, 

5 Riccio et. al (2013). 
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food stamps, and Safety Net Assistance (SNA) programs in New York City. Rewards payments 
were not counted as income or assets when determining eligibility for these benefits. 6 Thus, the 
program would have only affected benefit receipt if it had encouraged participants to move into 
employment or earn higher wages. As shown in Appendix Table C.l, Family Rewards 2.0 
overall had no effect on TANF or SNA benefit receipt — on either the likelihood of receiving 
payments or the average dollar amount received. 7 Similarly, program and control group mem- 
bers were equally likely to receive SNAP benefits, but program group members received higher 
SNAP payments than their control group counterparts, especially in Year 1 of follow-up — a 
statistically significant difference of $276. Appendix Table C.2 presents these results by site, 
showing that the effects on SNAP receipt were concentrated in Memphis. These effects might 
be related to the reduction in work also observed in Memphis (described in Chapter 7). 

Impacts on Banking and Financial Services 

Family Rewards 2.0 made payments electronically to families who earned rewards, depositing 
the money into bank accounts or onto stored value cards. Program staff encouraged families to 
open low- fee bank accounts in order to take advantage of the direct deposit of rewards, inviting 
banks that offered no-fee savings accounts to attend orientation sessions and market these 
accounts directly to families. Many families — almost 59 percent in New York City and 40 
percent in Memphis — did not have bank accounts when they entered the study and opted to 
open these new accounts. Direct deposit to bank accounts provided the program with an 
efficient mechanism for paying out cash rewards. In addition, linking families to mainstream 
financial institutions in this way had the potential to spur other positive effects, such as increas- 
ing savings and reducing the use of alternative services such as check cashiers or “payday 
loans,” 8 often available to low-income households at very high interest rates and in ways that 
many experts contend mislead and exploit the poor. 

Table 3.2 shows that almost 66 percent of the program group reported having a bank account at 
the time of the 24-month survey, compared with 44.4 percent of the control group respondents, 


6 For the Bronx sample, the SNA program provides assistance to individuals and families who do not qual- 
ify for the time-limited federal TANF program. SNA payments may take the form of direct cash aid to 
beneficiaries or vendor payments (for example, rent payments to landlords) made on the SNA recipients’ 
behalf. 

7 Recall that self-reported TANF receipt was much lower among Bronx participants than Memphis partici- 
pants at study entry, even though the administrative data show similar levels of receipt during each year of 
follow-up for the two cities. These data are not inconsistent but indicate a higher degree of cycling onto and off 
of TANF rolls for the Bronx sample. This pattern is evident by the fact that quarterly TANF receipt rates were 
much lower than yearly receipt rates for the Bronx sample. 

S A payday loan is a loan from a check-cashing outlet or other lending institution that must be repaid by the 
next payday. Payday loans are illegal in New York, but may be available through the Internet. 
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Table 3.2 

Impacts on Banking, Savings, and Debt 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Use of banking/financial services (%) 

Currently has any bank account 

65.5 

44.4 

2i i *** 

0.000 

Currently has checking account 

44.2 

35.2 

g q *** 

0.000 

Financial transactions at least once a month 

Cash check at check casher 

18.4 

22.8 

_4 4 ** 

0.017 

Pay bill at check casher 

26.3 

27.6 

-1.3 

0.496 

Family savings and debt 

Average savings ($) 

145 

82 

63 ** 

0.012 

$0 (%) 

79.5 

88.0 

-8.5 *** 

0.000 

Any (%) 

20.5 

12.0 

8.5 *** 

0.000 

$1 to $250 

7.9 

4.9 

3 Q H=H=H= 

0.007 

$251 to $500 

4.3 

2.1 

2 2 *** 

0.006 

More than $500 

6.2 

3.5 

2 2 *** 

0.006 

Average debt ($) 

7,308 

7,012 

295 

0.636 

$0 (%) 

43.6 

41.5 

2.1 

0.344 

$1 to $1,000 (%) 

8.6 

7.2 

1.4 

0.250 

$1,001 to $5,000 (%) 

18.0 

18.1 

-0.1 

0.971 

$5,001 to $15,000 (%) 

14.8 

20.0 

2 *** 

0.003 

More than $15,000 (%) 

14.2 

11.9 

2.3 

0.133 

Sample size (total = 2,016) 

1,025 

991 




SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for 
multiple observations per family. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 


a statistically significant difference of 21.1 percentage points. As noted in Chapter 2, more than 
96 percent of families in the program group earned a reward in Year 2 and received payments 
through direct deposit to their bank accounts (since a very small fraction of participants opted to 
receive these payments via stored value cards). It is possible that some enrolled families had 
bank accounts that closed from time to time or changed their bank accounts during the study 
period, so that at any given moment a portion of the participants may have not had a functional 
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bank account. 9 Program group members were also more likely (by 9 percentage points) to 
report having a checking account than control group members. Family Rewards 2.0 thus clearly 
generated a net increase in participants’ connectedness to mainstream banking institutions. 

Fewer program group members reported having used a check casher to cash a check in 
the month before the 24-month survey (18.4 percent of the program group compared with 22.8 
percent of the control group), a statistically significant difference of 4.4 percentage points. As 
noted above, Family Rewards 2.0 had a positive effect on linking families to mainstream 
financial institutions, which may have reduced their reliance on high-cost financial transactions 
through check cashers. 

Savings and Debt 

Families could use the cash rewards offered through Family Rewards 2.0 as they 
pleased — in other words, the program did not place any restrictions on how these payments 
should be spent. The program designers believed that imposing restrictions (such as limiting the 
use of rewards for education, training, or home ownership, as some asset-building programs do, 
or for any other particular purpose) would have reduced the rewards’ real incentive value. 
Consequently, an important question is whether, without such restrictions, the families used the 
additional income from the program to build their financial security. Previous studies suggest 
that resources received in large, lump sums, rather than in smaller, more regular payments, are 
more likely to be saved, used to pay down debt, or invested in human capital. 10 

At the 24-month point, about 2 1 percent of the program group reported having some 
family savings, compared with 12 percent of the control group, a statistically significant 
difference of 8.5 percentage points. The program also increased average savings. Program 
group members reported $145 in average savings, compared with $82 by the control group, a 
difference of $63. 11 

While the 24-month survey asked respondents if they or their spouses or partners had any 
money or savings, it was unclear whether the substantial earnings by student participants were 
contributing to family savings. Nonetheless, the in-depth qualitative interviews suggest that 
parents actively encouraged their children to save. One respondent, for example, said, “[y]ou 
know, my children need to save money, instead of spending it in one day. Save money, all 


9 A similar pattern was observed in Family Rewards 1.0. 

10 Chamber and Spencer (2008); Smeeding, Phillips, and O’Connor (2000). 

1 'The averages are for full program and control group samples, respectively, not just for those study group 
members with any savings. The impact on enrolled families’ savings early in the study period was higher in 
Family Rewards 1.0 ($221), but the families in that program were offered and earned more rewards. 
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right? Put $5.00, leave $5.00 in there, as much as you can. And then you’ll see it, like June 
comes, let’s see how much you saved up between now and June.” 

Related to the broader question of how families used the cash rewards is the question of 
whether the additional money was used to pay off debt. Exploring this potential effect in Family 
Rewards 2.0, the 24-month survey asked respondents to indicate whether they had any debt and, 
if they had, the amount owed. 12 Table 3.2 shows that both study groups reported relatively high 
amounts of family debt, with no difference on average ($7,308 for the program group versus 
$7,012 for the control group). 

Across the range of indicators that this report examines, it appears that Family Rewards 
2.0 had a positive impact on families’ financial security. Families in the program group were 
more likely to maintain bank accounts and have savings. Given that both groups reported 
similar levels of debt, despite the program’s modest effect on savings, it is possible that families 
used much of their extra earnings — and potential savings — to address immediate household 
needs, hardships, and help improve their quality of life. The next section examines these issues. 


Impacts on Material Hardship and Well-Being 

An important question about Family Rewards 2.0 is whether the program’s cash transfers and 
positive effects on household income translated into reduced hardship and improved well-being. 
This section examines a range of indicators that are likely to be affected by changes in house- 
hold income. Given that families in Family Rewards 2.0 were offered and earned smaller 
amounts of money from rewards than families in the original program, and given that a signifi- 
cant portion of the rewards were earned by students, it is an open question whether the program 
led to notable effects on material well-being. 

Unfortunately, there is neither a common definition of material well-being, nor is there 
a standard approach to its measurement. Most efforts to measure this concept include scales or 
indicators of various dimensions of need — or hardships — across domains such as housing 
and utilities, financial strain, and unmet needs. Drawing on a comprehensive set of measures, 
the 24-month survey examined how families were faring at the time of the interview or in the 
recent past. 


'“The survey questions on savings and debt were largely framed around family finances, and it is likely 
that respondents were reporting debt accumulated by the family rather than simply the respondents’ personal 
debt. 
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The top panel of Table 3.3 focuses on housing- and utilities-related hardships that fami- 
lies experienced in the 12 months before the time of the survey. Family Rewards 2.0 had little 
effect on any of the measures used to capture housing-related hardships. Both study groups 
reported similar experiences with regard to the housing-related hardships that the survey 
measured — for example, not paying their full rent or mortgage or being evicted for not paying 
their rent or mortgage. 13 

Related to hardship, the survey asked respondents a series of questions related to un- 
met needs. Large numbers of program and control group respondents responded affirmatively 
to the question of whether they ever did not have enough money to buy food sometime in the 
past 12 months: 43.3 percent of the program group reported this hardship, compared with 
47.3 percent of the control group, a statistically significant 4 percentage point reduction in this 
hardship for program group families. No notable differences were seen on measures of unmet 
health care needs, such as being unable to receive needed medical care because of cost or 
being unable to fill prescriptions for the same reason. As described later in this report, health 
coverage rates were very high for the study sample, which may have assured families’ access 
to needed health care. 

To assess respondents’ perceived financial circumstances more broadly, the survey 
asked respondents to compare their current financial situation with what it had been a year 
before on a financial strain scale. Exactly 60 percent of the program group reported that their 
current financial situation was better than it had been the year before, compared with 51.7 
percent of the control group, a statistically significant increase of 8.3 percentage points. Pro- 
gram group members also scored higher on a financial well-being scale than did control group 
members (9.3 points versus 9.1 points), indicating they had a more positive view of their 
financial well-being. The reader should keep in mind that the survey was conducted during Year 
2, when program group families were still earning cash rewards. 

The final set of measures shown in Table 3.3 captures three dimensions of psychologi- 
cal well-being: belief in one’s ability to initiate and sustain action, 14 perceived life satisfaction, 


l3 On a more technical note, the survey captured the absolute presence or absence of a set of material hard- 
ships in a 12-month period and did not ask about the frequency or intensity of the hardships experienced, which 
the cash rewards may have affected. 

14 The research team used the State of Hope scale to measure “agency,” or whether or not a person feels 
motivated to initiate and sustain action (for example, goal-directed thinking). The team took the listed items 
from Snyder et al. (1996). The Hope Scale scores correlate positively with measures of dispositional 
optimism, self-esteem, and problem-solving ability, and correlate negatively with measures of depression 
and hopelessness. 
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Table 3.3 


Impacts on Material Hardship, Financial Strain, and Psychosocial Well-Being 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Effect 

Size 

Any housing/utilities material hardship in 
the past 12 months (%) 

64.4 

64.1 

0.3 

0.879 


Did not pay full rent or mortgage 

41.4 

42.0 

- 0.6 

0.791 


Evicted from home for not paying 
rent or mortgage 

5.5 

5.4 

0.2 

0.864 


Did not pay foil utility bill 3 

42.6 

42.9 

-0.3 

0.876 


Utility was turned off 3 

13.3 

11.8 

1.5 

0.293 


Phone service was disconnected * 3 

22.0 

23.6 

- 1.6 

0.400 


Food security (1 = high; 4 = low) c 

3.2 

3.2 

0.0 

0.495 

0.030 

Insufficient food d (%) 

25.4 

26.7 

-1.3 

0.497 


Strongly or somewhat agree with 
the following (%) 

Financial situation is better than last year 

60.0 

51.7 

g 3 *** 

0.000 


Do not worry about having enough 
money in future 

20.7 

23.3 

- 2.6 

0.160 


Can generally afford to buy needed things 

70.9 

66.2 

47 ** 

0.023 


Sometimes have enough money to buy 
something or go somewhere just for fun 

30.8 

31.7 

-0.9 

0.665 


Financial well-being (4 = low; 16 = high ) 1 

9.3 

9.1 

0.2 ** 

0.021 

0.103 

Did not have enough money to buy food 
sometime in the past 12 months (%) 

43.3 

47.3 

-4.0 * 

0.074 


Did not get needed medical care because 
of cost in past 12 months f (%) 

11.1 

11.1 

- 0.1 

0.974 


Did not fill prescription because 
of cost in past 12 months (%) 

18.5 

19.5 

- 1.0 

0.573 


Psychosocial well-being 

Average score on "State of Elope" scale 
(6 = low; 24 = high ) 8 

17.8 

17.6 

0.2 ** 

0.042 

0.090 

Flow life today compares to way it was 
a year ago (%) 

Much or somewhat better 

66.4 

58.3 

g 2 *** 

0.000 


Level of happiness * 1 (%) 

Very or pretty happy 

76.2 

72.2 

4 o ** 

0.037 


Sample size (total = 2,016) 

1,025 

991 





(continued) 
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Table 3.3 (continued) 


SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics of sample members. Standard errors were adjusted to account for multiple observations per 
family. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The effect size is the difference between program and control group outcomes expressed as a proportion of 
the standard deviation of the outcomes for both groups combined. 

“Utilities include gas, oil, and electricity. 

b This outcome includes cellular or land service. 

c The food security question describes food eaten by the family in the prior month: 1 = Enough to eat of the 
kinds of food desired; 2 = Enough to eat but not always the kinds of food desired; 3 = Sometimes not enough 
to eat; 4 = Often not enough to eat. 

insufficient food is defined as "sometimes" or "often times" not having enough food to eat. 

Components of the financial well-being scale have been coded such that a lower score implies being 
worse off and a higher score implies being better off. The scale was calculated by summing responses to the 
four component questions, each of which is scored on a four-point scale. Thus, the financial well-being scale 
presented here ranges from 4 to 16 points. 

r This outcome excludes prescriptions. 

sThe "State ofFIope" scale measures the level of ongoing goal-directed thinking. The response codes (1 to 
4) of the six items for each person are summed, with lower values representing less goal-directed thinking 
and higher values representing more. The scale was taken from Snyder et al. (1996). 

h Happiness is measured using the U.S. General Social Survey question: "Taken all together, how would 
you say things are these days — would you say that you are very happy, pretty happy, or not too happy." 


and a global level of happiness. 15 It is possible that programs that involve regular cash transfers 
can reduce stress and bring about improvements in individual emotional and psychological 
well-being. There is well-established literature supporting a positive correlation between life- 
satisfaction measures and income, and one theme in the literature on happiness is the idea that 
life satisfaction may depend on relative rather than absolute income. 16 

The data show consistently positive effects on all three measures of psychosocial well- 
being. Program group members rated themselves higher on the State of Hope scale, were more 
likely to report that life was better than a year ago, and were also more likely to rate themselves 
as being pretty happy. Although Family Rewards 2.0 increased household income, the impacts 
were modest (a gain of $138, or 9.2 percent), and despite the income gains, program and control 
group participants experienced similar levels of material hardship — in other words, the cash 
transfers allowed for some minor reductions in hardship, but did not lead to notable changes in 
material well-being. 


l5 The 24-month survey used the same item that has been used on the U.S. General Social Survey. 
16 Lachowska (2015). 
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In what ways could Family Rewards 2.0 have made study participants feel happier and 
better about themselves? In-depth interviews conducted with a small number of program group 
families offer a few insights into the lives of study participants and the direct and indirect ways 
in which the cash transfers could have had an impact on their sense of well-being. 17 

At the most basic level, families enrolled in Family Rewards 2.0 felt supported in the 
program and viewed their advisors as a core source of support in their sometimes unpredictable 
lives. As one participant stated: 

They’re not there physically to help you, but. . .emotionally, they are. Yeah, and 
if you do need them, they’re a phone call away. You understand? Like I said, and 
it helps me and I just love it and the people that work here that I work with is 
great. Every time I call, it’s a great feeling. You can feel comfortable talking 
with them. 

The in-depth interviews also revealed a perspective common among participants that 
the program — with its incentives — functioned as an important parenting tool, something that 
motivated their children to work harder and do better at school. As children perfonned better in 
school, parents may have experienced some relief and reduced stress as they had one less 
parenting concern. As one participant put it: 

Oh hell yeah. They got him up and out of the house to school. That’s my reward. 

But his grades improved, his attendance has improved and he’s actually opening 
up books. The first day of high school, [I said] “where’s your books?” He’s like, 

“Oh, I leave them at school.” “So how you supposed to study?” That’s my re- 
ward. I don’t care about the cash or anything. My reward is that my son im- 
proved in his grades and so forth, so on. 

The extra money families earned from rewards allowed parents to spend more freely on 
themselves and their children, as they wished. The program designers had hoped that families 
would use the extra money in ways that align with program goals. Drawing on the 24-month 
survey, Table 3.4 presents parents’ accounts of how they used the cash rewards they earned. 
Parents most commonly used the rewards they earned to cover regular expenses (such as rent, 
food, and utilities) and to purchase consumer goods or special treats for the family (such as 
eating out). With respect to the rewards that students earned, most parents reported that they 
used those earnings to buy little extras (such as eating out, going to a movie, or buying clothes) 
and student-focused investments, and not for households expenses such as rent and utilities 
(although about a fourth of the parents also acknowledged that they used the education rewards 
to help cover basic expenses). Based on their responses, it appears that parents tried to balance 


n These insights are suggestive and are not meant to explain causal links. 


52 



Table 3.4 


How Families Used Rewards, Parent Survey Responses 
24 Months After Random Assignment 


Parent Child 

Bank Bank 

Use Family Rewards Payments (%) Account Account 


Help pay for regular expenses, such as rent, utilities, or food 65.6 23.8 

Pay off bills, such as credit cards or medical bills 29.8 5.8 

Make a major purchase, such as a house, major appliance, or car 7.9 2.4 

Save for some future need, such as college tuition or retirement 15.0 23.3 

Pay for health care, dental care, or health insurance 8.2 3.2 

Pay for things to help children in school, such as special lessons or private schools 19.9 16.2 

Pay for a few extras, such as eating out, going to a movie, or buying electronics or clothes 44.4 71.0 

Help other family members or friends with expenses 6.6 5.5 

Pay for college tuition or other costs for training 8.1 8.2 


Sample size 1,025 


SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: This table excludes control group members because it pertains only to the Family Rewards program. 
Sample sizes may vary because of missing values. 

Percentages may sum to more than the number participating in any activity because sample members 
could list more than one response. 

their responsibility to provide for the basic needs of their family with a desire to celebrate their 
achievements. For many families, this celebration took the form of a family activity such going 
on a trip or to a movie that would have otherwise been prohibitively expensive, especially for a 
large family with limited means. 

Thus, across different measures, the cash rewards themselves and the supportive rela- 
tionships with advisors, may have moderated and alleviated some of the stress that low-income 
parents commonly experience, which may have been reflected in participants’ improved sense 
of well-being. 

Conclusion 

This chapter examined Family Rewards 2.0’s short-term effects on household poverty and well- 
being. For the most part, the results support a CCT program’s hypothesized effects. During the 
period in which enrolled families supplemented their household income with reward payments, 
the program produced modest gains in household income; reduced the proportion of families 
who were living in poverty, including severe poverty; increased the proportion of families with 
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household savings, and reduced reports of financial strain. It also consistently produced positive 
effects on multiple dimensions of perceived well-being. There is less evidence, however, that 
the program reduced more extreme types of hardship measured in this study, which may have 
required more sizeable cash transfers. Finally, the research team examined variation in impacts 
across a range of baseline characteristics, including poverty status at random assignment. As 
Chapter 7 will show, effects on income and poverty were fairly consistent across several types 
of families. 

It is important to recall that, unlike Family Rewards 1.0, Family Rewards 2.0 targeted 
families receiving TANF and food stamps. This iteration of Family Rewards also offered fewer 
benefits (8 instead of 22), which meant the total amount that families in the program could have 
earned was less than the amount that families enrolled in its predecessor could have earned. 
However, Family Rewards 2.0 offered rewards monthly, potentially making the cash transfers 
more salient to families’ monthly budgets and material well-being. One other design modifica- 
tion relevant to understanding the program’s potential to affect material well-being was that the 
education rewards earned and paid directly to students made up a significant portion of total 
family earnings but were not always used toward meeting basic household needs. 

Despite several important design modifications, the findings from the Family Rewards 

2.0 evaluation presented in this report are largely consistent with those from Family Rewards 
1.0. For example, in the short temi, both interventions improved enrolled families’ economic 
and financial well-being, reduced poverty, linked families to mainstream banking institutions, 
and increased average family savings. One notable difference is that unlike in Family Rewards 

1.0 there was no evidence from the 24-month survey that families in Family Rewards 2.0 
experienced less housing-related hardship. While the extra income from the rewards helped 
families in both programs improve their material well-being, Family Rewards 2.0 was less 
likely than Family Reward 1.0 to affect more extreme types of hardships, perhaps because 
families in Family Rewards 2.0 earned less than their counterparts in Family Rewards 1.0. 
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Chapter 4 

Impacts on Education 


Family Rewards aimed to improve students’ school outcomes by providing students with 
incentives for meeting important milestones, including good grades, high attendance, and 
passing high school exit exams. The original program (Family Rewards 1.0) improved school 
performance and graduation rates for academically proficient ninth-graders. The new program 
(Family Rewards 2.0) targeted families with children entering ninth or tenth grade and was 
modified in the hopes that the changes would allow it to have effects on all students, even those 
struggling academically. 1 This chapter presents findings over four years on students’ progress in 
school, for the full sample of ninth- and tenth-graders and for subgroups defined by academic 
proficiency. In sum, the findings show that Family Rewards 2.0 had few lasting effects on 
students’ school outcomes, either for the full sample of students or for the more academically 
proficient subgroup. 


The Education Rewards 

Students could earn education rewards through four activities. First, they could receive $40 per 
month for maintaining high attendance, defined as 95 percent or higher. Second, they could earn 
$50 for taking college entrance exams, either the SAT or the ACT. In Memphis, students 
routinely take the ACT as part of school district policy. Thus, students earned the rewards for 
scoring above a minimum threshold. 2 Third, students could earn money for passing grades on 
report cards, with $10 for each C, $20 for each B, and $30 for each A. Finally, students could 
earn substantial rewards for taking and passing high school exit exams. In the Bronx, students 
could earn $500 for a score of 75 or higher (or $400 for a score between 65 and 74) on each of 
the five New York State Regents exams required to graduate. 3 In Memphis, they could earn 
$300 for passing each of the seven End-of-Course exams administered by the school district. 

Family Rewards 2.0 differed from Family Rewards 1.0 in several ways. First, as men- 
tioned in Chapter 1 , it offered fewer rewards in an effort to streamline the program and make it 
simpler for families to understand. In the case of education rewards, Family Rewards 2.0 did not 
include rewards for credits earned, for example, for graduation, or for parents to attend parent- 

'The program targeted entering ninth- or tenth-graders since they would have at least three years of expo- 
sure to the rewards. 

2 lf they took the exam independently of the school system, any score was rewarded. 

’Students are only required to pass these exams with a score of at least 65 in order to graduate. However, 
achieving a score of 75 or higher will exempt students from remediation at many New York colleges and is 
used as an admissions cutoff by some universities. 
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teacher conferences. Second, it structured the rewards in such a way as to make them more 
salient to families and children, by paying out benefits monthly rather than every two months, 
and by offering students rewards for grades on report cards. The program designers hoped that 
the rewards for grades, in particular, would be more salient and present in the minds of students 
as they attended and performed in class from week to week. The program also paid all of the 
rewards directly to students’ accounts, whereas some education rewards students earned in 
Family Rewards 1.0 were paid to parents’ accounts. During the first year, students had to visit 
an advisor in order to receive the grade rewards, which meant more interaction with the pro- 
gram. 4 In contrast, the rewards for high school exit exams were paid automatically and did not 
require a meeting with an advisor. While still important, these automatic and nonrecurring 
rewards were forgotten by many students in Family Rewards 1.0 during the course of the year. 
Finally, although not specific to the education domain, a big change from the original model 
was the more active guidance provided to families. If a child was struggling to improve school 
perfonnance, the advisor was expected to work with the family to refer the student to tutoring or 
other relevant services. 

As described in Chapter 2, each enrolled family earned about $1,200 in Year 2 from the 
education rewards, representing just over 50 percent of the total amount earned from all three 
domains. Over 90 percent of families earned at least one education reward, with the highest 
receipt rates for high attendance and report card grades of C or higher. 


Summary of Impacts on School Progress 

This section presents impacts for ninth- and tenth-grade students in Memphis and the Bronx 
combined. (As shown in Chapter 7, the findings are generally similar across both cities.) In 
addition, it presents impacts for ninth-graders only, in order to compare the findings more 
directly to those from Family Rewards 1.0. The research team obtained data on key education 
outcomes from administrative records provided by the New York City Department of Education 
(DOE) and the Memphis City Schools. These data were available for all students in the study 
for roughly one year before study entry, or school year 2010-201 1 , and for four years after study 
entry, or for school years 201 1-2012 through 2014-2015. As noted in Chapter 1, the follow-up 
years do not correspond exactly to post-enrollment years, since some families entered the study 
well into the first school year. For example, over 30 percent of families entered the study in 
January or February 2012, meaning that their children could not have responded to the rewards 


4 The mandate to meet with an advisor for every report card review was lifted in Year 2, in response to 
difficulties students faced in getting to the NPOs. However, as noted in Chapter 2, NPOs in the Bronx thought 
that these face-to-face meetings were important and did not announce that this requirement was dropped in 
Year 2. 
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until the second half of that school year. Thus, the second and third school years provide a more 
complete account of the program’s effects within a given year. 

The DOE and Memphis City Schools data include enrollment status, attendance rates, 
credits earned, and performance on Regents exams (in the Bronx) and End-of-Course exams (in 
Memphis). These data do not provide infomiation for students attending parochial schools, 
private schools, or schools outside of New York City or Memphis, but the survey data indicate 
that few students in the sample (about 5 percent) attended these other types of schools. The 
survey provides infonnation on intermediate outcomes, such as parental effort (parents’ interac- 
tion with their children and their children’s teachers), children’s engagement in extracurricular 
activities, and parents’ ratings of their children’s school performance. 

In sum, Family Rewards 2.0 had few effects on important markers of school progress 
through Year 4. The research team examined several outcomes, all of which are predictive of 
high school graduation, including whether students were “on grade” in a given year, or in the 
grade they should have been if they had progressed each year, attendance rates, credits earned, 
and the number of high school exams passed. The program did not have effects on these 
outcomes for the full sample of entering ninth- and tenth-graders, or for the subset of students 
who were more academically prepared. (Recall that the main effects of Family Rewards 1.0 
were for proficient ninth-graders.) When the analysis focused only on entering ninth-graders, 
there were small, positive effects observed on attendance and credits earned. However, these 
effects did not persist and did not lead to increased enrollment rates or graduation rates. The 
next sections present effects on these main outcomes and several secondary outcomes. 


Impacts on Attendance, Credits, and High School Exams 

Table 4.1 presents effects on several outcomes for students who entered the study as ninth- or 
tenth-graders. All of the outcomes, with the exception of graduation, cover three years, since 
many entering tenth-graders would not have been in school during Year 4. Graduation, in 
contrast, was measured through Year 4, capturing entering tenth-graders who took an extra year 
to finish school. 

The first panel presents enrollment status. Data for the control group show that 78 per- 
cent of entering ninth- and tenth-graders were on grade in Year 3. Of those students not on 
grade (22 percent), most were still in school but in a lower grade (12 percent). The remaining 10 
percent were no longer enrolled in either a Memphis or New York City public school. Survey 
data indicate that most of these students were attending other schools and had not dropped out. 
Family Rewards had no effects on these measures of school progress. Similarly, the program 
had no effect on graduation rates, as shown in the second panel. 
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Table 4.1 


Impacts on Enrollment, Graduation, Attendance, Credits, 
and Regents or End-of-Course Exams for Students in Grades 9 and 10 
at the Time of Random Assignment 



Program 

Control 

Difference 


Grade Level and Outcome 

Group 

Group 

(Impact) 

P- Value 

Enrollment, Year 3 (%) 





On grade 

79.4 

78.2 

1.2 

0.456 

Not on grade 

11.7 

11.8 

-0.1 

0.908 

Not enrolled 

8.9 

9.9 

-1.0 

0.368 

Graduation (%) 





Graduated on time 

63.4 

63.1 

0.3 

0.855 

Graduated by Year 4 a 

66.1 

66.1 

0.0 

0.995 

Attendance 0 (%) 





Attendance rate, Y ear 1 

86.9 

87.4 

-0.4 

0.578 

Attendance rate, Y ear 2 

82.5 

80.5 

2.0 * 

0.057 

Attendance rate, Y ear 3 

77.5 

75.8 

1.7 

0.162 

Attendance rate is 95% or higher, Year 1 

44.3 

43.5 

0.8 

0.685 

Attendance rate is 95% or higher, Year 2 

36.8 

35.9 

0.9 

0.645 

Attendance rate is 95% or higher, Year 3 

28.9 

29.1 

-0.2 

0.910 

Credits 0 





Number of credits earned, Year 1 

10.9 

11.0 

-0.1 

0.704 

Number of credits earned, Year 2 

10.0 

9.9 

0.1 

0.568 

Number of credits earned, Year 3 

9.1 

8.9 

0.2 

0.415 

Number of credits earned, Years 1 to 3 

30.1 

29.9 

0.2 

0.651 

Earned adequate credits, Years 1 to 3 (%) 

55.4 

55.0 

0.4 

0.833 

Regents or End-of-Course exams Q,e 





State core exams taken, Y ears 1 to 3 

6.4 

6.3 

0.1 

0.344 

State core exams passed, Years 1 to 3 

2.7 

2.7 

0.0 

0.716 

Sample size (total = 2,676) 

1,343 

1,333 



(continued) 
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Table 4.1 (continued) 


SOURCES: MDRC calculations using data from New York City Department of Education and Shelby 
County Schools administrative records. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Note that all outcomes in the table include zero values for students who were no longer enrolled. 

Years 1 , 2, 3 , and 4 cover the 20 1 1 -20 1 2, 20 1 2-20 1 3 , 20 1 3 -20 1 4, and 20 1 4-20 1 5 school years, 
respectively. 

a Students enrolled in tenth grade at the time of random assignment had five years to complete 
graduation in this measure. Students enrolled in ninth grade at the time of random assignment had four 
years. 

b Attendance was calculated as a percentage of total days present divided by total days enrolled 
according to district records. Records provided for students in New York City include enrollment for 
the regular school year. Records for students in Memphis include enrollment during the regular school 
year, alternative education programs, and summer school. 

c Students in New York City earn 1 credit per course per semester completed. Students in Memphis 
earn 0.5 credits per course per semester. Credits for students in Memphis were multiplied by two to 
create a standard scale for comparison. To be considered on time to graduate, students in New York 
City must earn an average of 1 1 credits per school year and students in Memphis must earn an average 
of 5.5 credits per school year. 

d The Regents exam measures in this table include the following Regents exams: English, Math A, 

Math B, Geometry, Integrated Algebra, Algebra 2/Trigonometry, U.S. History and Government, 

Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

e The End-of-Course exam measures in this table include the following exams: English 1, English, 

2, English 3, Biology, Algebra 1, Algebra 2, and U.S. History. 

The third panel presents data on attendance. The program had no overall effect on at- 
tendance, with the exception of a small, positive impact on the attendance rate in Year 2. A 
minority of control group students had high attendance in any given year — only 36 percent in 
Year 2 and 29 percent in Year 3. As noted in the table, all outcomes shown include zero values 
for students who were no longer enrolled in school. 

Although rewarded by the program, a high attendance rate may not give an accurate 
picture of students’ attendance. Chronic absenteeism, for example, is typically defined as less 
than 90 percent attendance. Only 66 percent of students in both research groups attended 90 
percent of school days in Year 1, indicating that just over a third of students were chronically 
absent (not reported in table). The program had no effect on the high attendance rate (as shown 
in Table 4.1) or on attendance of 90 percent or higher (not shown). 

The fourth panel in Table 4.1 presents data on credits earned during the year — the 
average credits students earned and whether the student passed an adequate number of credits 
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to stay on track for graduation. A student passing adequate classes to stay on track would earn 
11 credits per year in the Bronx and 5.5 credits in Memphis. To standardize the analysis, the 
research team multiplied the data on credits by two for Memphis students. The data show that 
although 78 percent of students were on grade in Year 3, only 55 percent of them were on 
track in terms of credits earned. By rewarding higher grades and attendance, Family Rewards 
2.0 might have affected credits earned. However, there are no impacts on either average or 
on-track credits. 

The final panel presents data on high school exit exams, or Regents exams in New York 
and End-of-Course exams in Temiessee. Students in New York must take and pass at least five 
Regents exams covering different subject areas (for example, math, English, science, and 
history) in order to graduate with a Regents diploma. They must take them before graduation, 
although most students take one or two each year of high school. Students in Memphis must 
take End-of-Course exams associated with each of seven core courses. Students typically take 
them after having completed the relevant course. Although it is possible to pass the course 
without passing the End-of-Course exam, the exam accounts for a large fraction of the final 
course grade. The data show that students on average had taken about six exams on average 
during the study period and had passed about three exams. Family Rewards 2.0 had no effect on 
the rate of taking or passing these exams. 

Although Family Rewards 2.0 rewarded grades, with a tiered reward structure for 
grades of A, B, and C, the impact analysis did not examine the effects on grades for two 
reasons. First, although it was reasonable to expect that the program might encourage some 
students to improve their grades, which should translate into earning more credits, grades by 
themselves are ultimately not crucial measures of students’ progress toward graduation. More 
predictive outcomes are attendance, credits earned, and grade progression. Second, using grades 
to calculate an overall grade point average might have provided a noisy picture of student 
progress, since it would have required making many assumptions about which course grades 
should contribute to this average and how much they should contribute. 

Table 4.2 presents Family Rewards 2.0’s effects on entering ninth-graders over four 
years. It displays impacts separately for this group, to match the analysis for Family Rewards 
1.0. In addition, although it is not obvious that the program’s effects would differ between 
ninth- and tenth-graders, ninth grade is an important transition year into high school, and these 
students might have been more responsive to the rewards. The findings show that the program 
led to small improvements in attendance and credits earned for the ninth-grade sample. Alt- 
hough encouraging, these effects were small and did not lead to effects on enrollment or 
graduation. Just over 60 percent of ninth-graders in the program group had graduated within 
four years, with a similar rate for ninth-graders in the control group. 
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Table 4.2 


Impacts on Enrollment, Graduation, Attendance, Credits, 
and Regents or End-of-Course Exams for Students in Grade 9 
at the Time of Random Assignment 



Program 

Control 

Difference 



Grade Level and Outcome 

Group 

Group 

(Impact) 


P-Value 

Enrollment and Graduation, Year 4 (%) 






On grade 

71.4 

69.7 

1.7 


0.482 

Not on grade 

10.4 

10.4 

0.0 


0.990 

Not enrolled 

18.2 

19.9 

-1.7 


0.408 

Graduated on time 

61.7 

60.7 

1.0 


0.701 

Attendance 8 (%) 






Attendance rate, Y ear 1 

86.7 

86.8 

-0.1 


0.921 

Attendance rate, Y ear 2 

82.4 

79.7 

2.7 

** 

0.041 

Attendance rate, Y ear 3 

79.2 

76.1 

3.1 

** 

0.040 

Attendance rate, Year 4 

61.8 

58.8 

3.0 


0.160 

Attendance rate is 95% or higher, Y ear 1 

45.0 

43.1 

1.9 


0.464 

Attendance rate is 95% or higher, Y ear 2 

37.0 

34.9 

2.1 


0.409 

Attendance rate is 95% or higher, Y ear 3 

33.0 

32.9 

0.1 


0.968 

Attendance rate is 95% or higher, Y ear 4 

17.4 

17.7 

-0.4 


0.854 

Credits' 5 






Number of credits earned, Y ear 1 

10.8 

10.7 

0.1 


0.763 

Number of credits earned, Y ear 2 

10.2 

9.7 

0.5 

* 

0.071 

Number of credits earned, Y ear 3 

9.5 

8.9 

0.6 

** 

0.044 

Number of credits earned, Y ear 4 

8.6 

8.3 

0.3 


0.279 

Number of credits earned, Years 1 to 4 

39.1 

37.6 

1.5 

* 

0.094 

Earned adequate credits, Y ears 1 to 4 (%) 

54.0 

53.1 

0.9 


0.728 

Regents/End-of-Course exams cd (%) 






State core exams taken, Y ears 1 to 4 

7.6 

7.2 

0.4 

* 

0.098 

State core exams passed, Years 1 to 4 

3.1 

3.1 

0.0 


0.976 

Sample size (total = 1,471) 

734 

737 




(continued) 
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Table 4.2 (continued) 


SOURCES: MDRC calculations using data from New York City Department of Education and Shelby 
County Schools administrative records. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Note that all outcomes in the table include zero values for students who were no longer enrolled. 

Years 1, 2, 3, and 4 cover the 2011-2012, 2012-2013, 2013-2014, and 2014-2015 school years, 
respectively. 

Attendance was calculated as a percentage of total days present divided by total days enrolled 
according to district records. Records provided for students in New York City include enrollment for 
the regular school year. Records for students in Memphis include enrollment during the regular school 
year, alternative education programs, and summer school. 

b Students in New York City earn 1 credit per course per semester completed. Students in Memphis 
earn 0.5 credits per course per semester. Credits for students in Memphis were multiplied by two to 
create a standard scale for comparison. To be considered on time to graduate, students in New York 
City must earn an average of 1 1 credits per school year and students in Memphis must earn an average 
of 5.5 credits per school year. 

c The Regents exam measures in this table include the following Regents exams: English, Math A, 
Math B, Geometry, Integrated Algebra, Algebra 2/Trigonometry, U.S. History and Government, Global 
History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

d The End-of-Course exam measures in this table include the following exams: English 1, English, 2, 
English 3, Biology, Algebra 1, Algebra 2, and U.S. History. The U.S. History exam was not 
administered for students in Year 4. 


Impacts on Parent-Child Interactions and Child Activities 

Family Rewards 1.0 had hoped to increase parents’ involvement in their children’s education 
through the education rewards in general, as well as through specific rewards for attending 
parent-teacher meetings and checking test results. The program did lead to small increases in 
parents’ involvement, as reported on surveys. It also led to an increase in engagement in 
extracurricular activities (such as sports, non-sports lessons, and tutoring) for children in middle 
school and high school. 

Although Family Rewards 2.0 did not reward parental involvement directly, it may 
have led to increased parent-child interactions about schooling through the general education 
rewards. Table 4.3 presents the effects on these interactions, using parents’ responses in the 
24-month survey. Overall, the findings suggest that the program had little effect in this area, 
given the small number of significant differences and their small size. For example, there are 
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Table 4.3 


Impacts on Parent-Child Interactions and 
Focal Child’s Educational Outcomes and Activities, 
High School Students 



Program Control Difference 


Effect 

Outcome 

Group 

Group 

(Impact) 

P-Value 

Size 

Parent-child interactions in past month 

Respondent has done the following 
(1 = never; 4 = several times per week) 

Talked with child about school 

3.7 

3.7 

0.1 

* 0.079 

0.081 

Helped child with homework 

2.4 

2.5 

0.0 

0.459 

-0.034 

Checked to see child's homework was complete 

3.3 

3.3 

0.0 

0.692 

0.018 

Helped child prepare for test 

2.4 

2.5 

0.0 

0.737 

-0.016 

Respondent discussed child's report card with child 

during 2012-2013 school year (%) 

95.8 

93.9 

1.9 

* 0.070 


Activities since random assignment (%) 

Child has done the following since random assignment 

Participated in educational programs, such as extra 

classes or tutoring 

60.5 

60.6 

-0.1 

0.950 


Enrolled in college exam preparation program 

66.1 

68.5 

-2.4 

0.264 


Received special education 

15.9 

15.2 

0.8 

0.572 


Prepared for the ACT or SAT 

84.9 

83.5 

1.4 

0.418 


Took the ACT or SAT 

57.7 

62.5 

-4.9 

** 0.026 


Child participated in any extracurricular activity 

87.8 

85.7 

2.1 

0.185 


Program to help with schoolwork, test preparation, or homework 

59.0 

59.4 

-0.5 

0.830 


Sports 

60.5 

57.2 

3.3 

0.141 


Non-sports lessons 

42.2 

42.6 

-0.4 

0.865 


Club or youth group 

46.5 

41.1 

5.4 

** 0.018 


Work for pay 

25.8 

22.6 

3.2 

0.107 


Sample size (total = 1,881) 

976 

905 





SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 


NOTES: This table presents outcomes only for focal children who were living in the household at the time of 
the survey and at random assignment. 

Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics of sample members. Standard errors were adjusted to account for multiple observations per 
family. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The effect size is the difference between program and control group outcomes expressed as a proportion of 
the standard deviation of the outcomes for both groups combined. 
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two significant impacts on parent engagement (talking with children about school and review- 
ing report cards). However, these effects were small and in addition to fairly high levels of 
engagement. 

The next panel presents effects on various activities, including college-preparatory and 
extracurricular activities. Family rewards 2.0 had no consistent effect on the five educational 
activities examined, with the exception of an odd negative difference for taking the ACT or 
SAT. Similarly, the results suggest little effect on extracurricular activities. 


Impacts for Proficient Versus Non-Proficient Students 

Although Family Rewards 1.0 had modest effects on ninth-graders overall, its main effects were 
for the subgroup of ninth-graders who were academically proficient when they entered the 
study. For students who were proficient in reading (or scored at the proficient level or higher on 
the English language arts exam), for example, the program led to sizable effects on enrollment, 
attendance, scores on Regents exams, and, ultimately, graduation. The program’s effect on the 
four-year graduation rate, an 8 percentage point increase, was especially notable and similar in 
size to other more intensive interventions. 5 The program designers modified the original model 
with the hopes that it would continue to affect school progress for proficient students but also 
begin to improve school outcomes for those students that were doing less well academically. 

This section presents effects for students classified by proficiency status at study entry. 
Using each district’s standards, the research team determined proficiency status by students’ 
scores on their eighth-grade standardized English or math tests. In the Bronx, 25 percent of 
students were determined to be proficient in English and 43 percent were proficient in math. 6 
Among students in Memphis, proficiency rates were 14 percent in English and 7 percent in 
math. 7 


Table 4.4 presents Family Rewards 2.0’s effects for entering ninth- and tenth-graders, 
by English proficiency test scores. As expected, data for the control groups show that students 


5 A longer-term analysis found that the program increased the six-year graduation rate for this group by 9.5 
percentage points (Riccio and Miller, 2016). 

6 ln Family Rewards 1.0, about 30 percent of entering ninth-graders had scored at the proficient level or 
higher on the English language arts (ELA) exams and 33 percent were proficient in math. 

7 The percentage of proficient students enrolled Family Rewards 2.0 was much lower in Memphis, com- 
pared with those in the Bronx and compared with the sample in the first program. To compare findings, a 
separate analysis examined impacts for the students who scored in the top third for the ELA exams, compared 
with students who scored in the bottom two-thirds. The findings (not shown here) similarly indicate few 
differences in impacts between the two groups. 
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Table 4.4 


Impacts on Enrollment, Graduation, Attendance, Credits, and Regents or End-of-Course Exams 
for Students in Grades 9 and 10 at the Time of Random Assignment, by English Proficiency Test Score 


Proficient on Eighth-Grade ELA Test Not Proficient on Eighth-Grade ELA Test 


Grade Level and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Enrollment, Year 3 (%) 

On grade 

89.9 

91.8 

-2.0 

0.446 

80.7 

78.3 

2.4 

0.186 

Not on grade 

7.5 

3.3 

4 | ** 

0.041 

12.0 

13.5 

-1.5 

0.331 

Not enrolled 

2.7 

4.8 

-2.2 

0.223 

7.3 

8.3 

-1.0 

0.439 

Graduation (%) 

Graduated on time 

80.9 

79.4 

1.4 

0.700 

62.5 

62.0 

0.6 

0.802 

Graduated by Year 4 J 

81.7 

81.9 

-0.2 

0.966 

64.9 

65.0 

-0.2 

0.934 

Attendance 15 (%) 

Attendance rate, Year 1 

90.3 

88.4 

1.8 

0.363 

87.2 

88.4 

-1.2 

0.148 

Attendance rate. Year 2 

88.2 

86.5 

1.7 

0.418 

82.7 

81.4 

1.3 

0.256 

Attendance rate, Year 3 

85.7 

83.8 

2.0 

0.372 

78.5 

77.0 

1.5 

0.264 

Attendance rate is 95% or higher, Year 1 

57.6 

54.6 

3.0 

0.499 

42.4 

42.8 

-0.4 

0.866 

Attendance rate is 95% or higher, Year 2 

49.2 

48.5 

0.7 

0.870 

35.4 

34.5 

0.9 

0.699 

Attendance rate is 95% or higher, Year 3 

37.9 

42.3 

-4.4 

0.309 

28.3 

27.2 

1.1 

0.585 

Credits'^ 

Number of credits earned, Year 1 

12.3 

12.3 

0.0 

0.935 

10.8 

11.0 

-0.2 

0.458 

Number of credits earned, Year 2 

11.3 

11.4 

0.0 

0.964 

10.0 

9.9 

0.1 

0.799 

Number of credits earned, Year 3 

10.8 

10.6 

0.2 

0.670 

9.2 

9.0 

0.3 

0.302 


(continued) 



Table 4.4 (continued) 


Proficient on Eigth-Grade ELA Test Not Proficient on Eigth-Grade ELA Test 


Grade Level and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Number of credits earned, Years 1 to 3 

34.5 

34.3 

0.2 

0.858 

30.0 

29.8 

0.2 

0.762 

Earned adequate credits, Years 1 to 3 (%) 

71.4 

70.7 

0.7 

0.869 

54.3 

53.9 

0.4 

0.876 

Regents or End-of-Course exams d e 

Number of state core exams taken, Years 

1 to 3 

6.0 

6.2 

-0.2 

0.371 

6.7 

6.5 

0.2 

0.276 

Number of state core exams passed, Years 

1 to 3 

4.4 

4.7 

-0.3 * 

0.098 

2.4 

2.3 

0.1 

0.473 f 

Sample size (total = 2,433) 

225 

248 



1,008 

952 




SOURCES: MDRC calculations using data from New York City Department of Education and Shelby County Schools administrative records. 
NOTES: ELA = English language arts. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. 
Standard errors were adjusted to account for multiple observations per family. Statistical significance levels are indicated as follows: *** = 1 percent; 
** = 5 percent; * = 10 percent. Statistical significance levels across subgroups are indicated as follows: fff = 1 percent; f f = 5 percent; 
f = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Note that all outcomes in the table include zero values for students who were no longer enrolled. 

Years 1, 2, 3, and 4 cover the 2011-2012, 2012-2013, 2013-2014, and 2014-2015 school years, respectively. 

a Students enrolled in tenth grade at the time of random assignment had five years to complete graduation in this measure. Students enrolled in 
ninth grade at the time of random assignment had four years. 

b Attendance was calculated as a percentage of total days present divided by total days enrolled according to district records. Records provided for 
students in New York City include enrollment for the regular school year. Records for students in Memphis include enrollment during the regular 
school year, alternative education programs, and summer school. 

c Students in New York City earn 1 credit per course per semester completed. Students in Memphis earn 0.5 credits per course per semester. 

Credits for students in Memphis were multiplied by two to create a standard scale for comparison. To be considered on time to graduate, students in 
New York City must earn an average of 1 1 credits per school year and students in Memphis must earn an average of 5.5 credits per school year. 

d The Regents exam measures in this table include the following Regents exams: English, Math A, Math B, Geometry, Integrated Algebra, Algebra 
2/Trigonometry, U.S. History and Government, Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

e The End-of-Course exam measures in this table include the following exams: English 1, English, 2, English 3, Biology, Algebra 1, Algebra 2, 
and U.S. History. 



who were proficient on eighth-grade tests had better school outcomes than their less proficient 
counterparts — 92 percent of proficient students were on-grade in Year 3, for example, com- 
pared with 78 percent of non-proficient students. Similarly, proficient students had earned more 
credits and had passed more high school exams by the end of Year 3. The program, however, 
had no effects for either group of students. 

Table 4.5 presents Family Rewards 2.0’s effects for entering ninth-graders only, by 
English proficiency test scores. The table shows that the program had small, positive effects on 
attendance and credits earned for the less proficient group. However, these effects were not 
significantly different from the effects for the proficient students. In addition, the effects by 
proficiency status are not consistent. When the research team examined the impacts by math 
proficiency test scores, the results showed a pattern of more positive effects for students 
proficient in math, although many of the differences were not statistically significant. (See 
Appendix Tables D. 1 and D.2.) 


Conclusion 

Family Rewards 2.0 had few lasting effects on school progress for the target group of ninth- and 
tenth-graders. There is some suggestive evidence that ninth-graders may have responded more 
strongly to the program than tenth-graders, but overall the program still had no effect on their 
enrollment or graduation rates. The results by proficiency status were not conclusive and the 
results depended on whether proficiency was defined by reading or math. 

This finding stands in contrast to the large, positive findings for proficient ninth- 
graders in Family Rewards 1.0. Although it is it unclear why the new program had no effects, 
the results suggest that the effects of this type of conditional cash transfer on school progress 
were not very robust. One potential reason for the different effects is that the samples of 
proficient students were somewhat different between the two programs. First, in 2010, New 
York State made several changes to its standardized tests, making the tests more difficult and 
redefining the cutoff required for proficiency. As a result, the percentage of students in New 
York City that tested proficient dropped dramatically, meaning that proficient students in 
Family Rewards 2.0 might have been a more select group. However, additional analysis (not 
shown) that used a less stringent measure of proficiency found similar results. Second, the 
Family Rewards 2.0 results were for students from families receiving TANF or SNAP 
benefits in both Memphis and the Bronx, whereas the earlier findings were just for students in 
New York City who were eligible for the free lunch program. However, Family Rewards 2.0 
similarly produced no effects for proficient students when the sample was restricted to ninth- 
graders from families living in the Bronx. 
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Table 4.5 


Impacts on Enrollment, Graduation, Attendance, Credits, and Regents or End-of-Course Exams 
for Students in Grade 9 at the Time of Random Assignment, by English Proficiency Test Score 


Proficient on Eighth-Grade ELA Test Not Proficient on Eighth-Grade ELA Test 


Grade Level and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Enrollment and Graduation, Year 4 (%) 

On grade 

81.4 

86.7 

-5.3 

0.272 

70.5 

67.7 

2.8 

0.315 

Not on grade 

1.5 

2.5 

-0.9 

0.610 

12.4 

12.4 

0.0 

0.996 

Not enrolled 

17.1 

10.9 

6.2 

0.172 

17.1 

19.9 

-2.8 

0.230 

Graduated on time 

78.9 

83.5 

-4.6 

0.400 

60.0 

57.9 

2.1 

0.476 

Attendance 8 (%) 

Attendance rate, Year 1 

90.0 

89.8 

0.2 

0.944 

86.8 

87.2 

-0.4 

0.699 

Attendance rate, Year 2 

87.9 

86.2 

1.7 

0.590 

82.6 

79.9 

2.7 * 

0.070 

Attendance rate, Year 3 

86.7 

85.3 

1.4 

0.637 

79.3 

75.9 

3.4 * 

0.051 

Attendance rate, Year 4 

70.6 

70.0 

0.6 

0.902 

60.9 

57.4 

3.5 

0.153 

Attendance rate is 95% or higher, Year 1 

54.9 

56.7 

-1.9 

0.759 

43.9 

41.2 

2.7 

0.353 

Attendance rate is 95% or higher, Year 2 

50.9 

47.3 

3.6 

0.576 

35.3 

33.1 

2.2 

0.433 

Attendance rate is 95% or higher, Year 3 

42.8 

49.1 

-6.2 

0.322 

32.3 

29.5 

2.8 

0.296 

Attendance rate is 95% or higher, Year 4 

23.0 

23.9 

-0.9 

0.877 

17.1 

16.7 

0.4 

0.873 

Credits' 5 

Number of credits earned. Year 1 

12.3 

12.5 

-0.2 

0.792 

10.5 

10.4 

0.1 

0.698 

Number of credits earned. Year 2 

11.4 

11.5 

-0.1 

0.942 

10.1 

9.5 

0.6 * 

0.062 

Number of credits earned, Year 3 

10.8 

11.5 

-0.7 

0.309 

9.4 

8.6 

0.8 ** 

0.016 

Number of credits earned. Year 4 

9.8 

10.2 

-0.4 

0.606 

8.6 

8.2 

0.4 

0.238 
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Table 4.5 (continued) 


Proficient on Eighth-Grade ELA Test Not Proficient on Eighth-Grade ELA Test 


Grade Level and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Number of credits earned, Y ears 1 to 4 

44.3 

45.6 

-1.3 

0.576 

38.7 

36.7 

2.0 * 

0.059 

Earned adequate credits, Years 1 to 4 (%) 

68.0 

74.9 

-6.9 

0.246 

52.9 

50.1 

2.8 

0.347 

Regents or End-of-Course exams c,a 

Number of state core exams taken, Y ears 

1 to 4 

6.8 

6.8 

0.0 

0.995 

7.9 

7.5 

0.4 

0.120 

Number of state core exams passed, Years 

1 to 4 

5.2 

5.2 

0.0 

0.981 

2.7 

2.7 

0.1 

0.703 

Sample size (total = 1,391) 

115 

132 



588 

556 




SOURCES: MDRC calculations using data from New York City Department of Education and Shelby County Schools administrative records. 
NOTES: ELA = English language arts. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. 
Standard errors were adjusted to account for multiple observations per family. Statistical significance levels are indicated as follows: *** = 1 percent; 
** = 5 percent; * = 10 percent. Statistical significance levels across subgroups are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 
percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Note that all outcomes in the table include zero values for students who were no longer enrolled. 

Years 1, 2, 3, and 4 cover the 2011-2012, 2012-2013, 2013-2014, and 2014-2015 school years, respectively. 

Attendance was calculated as a percentage of total days present divided by total days enrolled according to district records. Records provided for 
students in New York City include enrollment for the regular school year. Records for students in Memphis include enrollment during the regular 
school year, alternative education programs, and summer school. 

b Students in New York City earn 1 credit per course per semester completed. Students in Memphis earn 0.5 credits per course per semester. 

Credits for students in Memphis were multiplied by two to create a standard scale for comparison. To be considered on time to graduate, students in 
New York City must earn an average of 1 1 credits per school year and students in Memphis must earn an average of 5.5 credits per school year. 

c The Regents exam measures in this table include the following Regents exams: English, Math A, Math B, Geometry, Integrated Algebra, Algebra 
2/Trigonometry, U.S. History and Government, Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

d The End-of-Course exam measures in this table include the following exams: English 1, English, 2, English 3, Biology, Algebra 1, Algebra 2, 
and U.S. History. The U.S. History exam was not administered for students in Year 4. 




Chapter 5 

Impacts on Health Care and Health 


Conditional cash transfer programs have been an important tool used in low- and middle- 
income countries to improve low-income families’ access to health care and preventive medical 
services. Historically, these programs have often featured incentives for regular checkups at 
health clinics, participation in health and nutrition education sessions, and engagement in other 
preventive health activities such as vaccinations or pre-natal care. 1 Some previous studies using 
health incentives have had positive effects on the receipt of health care services, as well as on 
certain health outcomes among poor families. 2 Family Rewards 2.0 incorporated these findings 
into its design and adapted incentives to support low-income families in the context of the 
health care system and public support available in the United States. The program offered 
rewards to low-income families for preventive medical and dental care visits. This chapter 
describes the health impacts of Family Rewards 2.0. In sum, the findings show that the program 
produced some modest effects on outcomes that are generally considered difficult to influence 
through social interventions. 3 However, the program did not produce effects on many other 
important measures. 


The Health Care Rewards 

Each family member in Family Rewards 2.0 was eligible to earn $300 each program year from 
two different rewards. 4 Individuals could earn $100 for completing a non-emergency check-up 
once per year and $100 for completing a regular cleaning and dental check-up twice per year. 
Family Rewards 1.0 offered two additional rewards for maintaining health insurance and for 

Femald, Gertler, and Neufeld (2008); Fiszbein, Schady, and Ferreira (2009). 

2 Giuffrida and Torgerson (1997); Kane, Johnson, Town, and Butler (2004); Volpp et al. (2008); Volpp et 
al. (2009). 

3 The literature on the effectiveness, or the relative value, of preventive clinical interventions is vast and 
inconsistent. In general, the response to preventive clinical interventions is weak. In a systematic review and 
analysis of recent interventions, insufficient evidence was found to support the effectiveness of a variety of 
counseling and preventive care practices; see Maciosek et al. (2006). While some evidence suggests that health 
insurance alone has beneficial effects on life expectancy, these impacts are small or conclusive only for some 
population groups; see Muennig, Franks, and Gold (2005) and Levy and Meltzer (2008). Recent evaluations of 
the Oregon Health Experiment found that access to Medicaid increased use of services and improved self- 
reported health status, but they did not find effects on biomarkers associated with physical health; see Finkel- 
stein et al. (2011). On the other hand, the effects of social interventions, such as improved early childhood 
education, have demonstrated effective ways of improving later health outcomes; see Belfield and Levin 
(2007). These interventions require many years to demonstrate effects. 

4 Children under 6 years of age were only eligible for one dental reward per year, making them eligible for 
$200 in rewards per year. 
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physician-advised follow-up visits. These rewards were both discontinued during the third year 
of that program, and were not included in Family Rewards 2.0. Increasing access to and use of 
preventive health care was expected to serve two primary functions: (1) to ensure that partici- 
pants received regular medical attention, which can prevent the escalation of health problems 
that can become more serious and costly when left untreated and (2) to decrease the use of 
costly medical services such as emergency rooms and hospital outpatient services by encourag- 
ing participants to establish a relationship with a primary care doctor who serves as a first 
responder. 5 As described in Chapter 2, a large number of families earned health rewards, with 
almost 90 percent of families earning at least one health reward during the program. Families 
that earned at least one health reward earned $1,805 on average in health rewards. 


Summary of Impacts on Health 

This section presents a summary of the effects of Family Rewards 2.0 on a variety of health- 
related outcomes. The research team obtained data on all outcomes from the 24-month survey 
and designated outcomes as primary (non-emergency medical visits, dental visits, and overall 
health status) or secondary (all others). The program produced impacts on several primary 
outcomes targeted by the program’s reward incentives. Both adult and child participants 
experienced a statistically significant increase in the frequency of dental visits. The program 
marginally increased adult participants’ self-rated health status by 0.1 on a 5-point scale; 6 this 
impact was much larger for adults who reported poor health at random assigmnent (0.3). The 
research team observed no statistically significant effects on any other primary outcomes. 


Impacts on Receipt of Health Care Services and 
Health Insurance Coverage 

The health-related cash incentives offered in Family Rewards 2.0 were directly connected to 
participants’ receipt of preventive medical and dental services and intended to create a direct 
increase in participants’ frequency of preventive service visits. 7 As a secondary effect, the 
program designers had hoped that an increase in preventive visits would also reduce reliance 
on costly services such as emergency room and hospital outpatient visits, especially for 
routine, non-emergency care, as participants established a relationship with a regular care 


5 See Chapter 1 and Riccio et al. (2010) for a discussion of the logic model and the pathways through 
which Family Rewards was expected to influence health outcomes. 

6 The research team assessed perceived health status based on an item drawn from the “Short Form 12 
Health Survey,” a widely used scale in national health surveys that provides a generic, multi-dimensional 
measure ofphysical and mental health; see Hays, Sherbourne, and Mazel (1995). 

7 Dechausay, Miller, and Quiroz-Becerra (2014); Riccio et al. (2010) 


72 



provider at a clinic, health center, or doctor’s office. 8 High insurance coverage rates for 
families left little room for Family Rewards 2.0 to improve connection to health care and 
preventive care practices. 9 

Table 5.1 presents outcomes for preventive care visits and health insurance coverage. 
Program group members did not experience a statistically significant change in annual health 
checkups. Parents in the program group were more likely than those in the control group to have 
had one or two dental checkups in the year before being interviewed (12.3 percentage points 
and 13.6 percentage points, respectively). 10 Many dentists in the United States are private 
practitioners and do not participate in Medicaid programs, 11 which can make regular dental 
visits costly and difficult to arrange, particularly for adults who are not guaranteed coverage 
under Medicaid. 12 The dental reward in Family Rewards 2.0 offset the costs of copayments and 
out-of-pocket expenses that families might have otherwise incurred. 13 A comparison of these 
data with reward receipt data shows that a higher percentage of participating adults reported 
having had both non-emergency medical and biannual dental checkups than did those who 
received rewards for these same activities. The process of documenting the visit and submitting 
a coupon to the program may have been a burden to some participants. 


8 By encouraging participants to have a regular health care provider, the program promoted the “medical 
home” model of health care delivery, which includes an ongoing relationship between a provider and patient 
and a comprehensive approach to care and coordination of care through providers and community services. 
Research suggests that individuals who have continuity of care with a regular practitioner are more likely to 
adhere to prescribed medications and to receive preventive care and well-coordinated, resource-efficient, and 
family-centered care, and are less likely to visit the emergency department and be hospitalized; in addition, 
their practitioners are more likely to recognize their problems and track their information. See Christakis et al. 
(2001); Christakis et al. (2002, 2003); Starfield and Shi (2004). 

9 Though Congress passed the Affordable Care Act six months before Family Rewards 2.0 began, it is 
unlikely that it greatly affected participants’ enrollment in health coverage because many families were already 
enrolled in public health insurance at program enrollment. The 24-month survey also indicated high levels of 
enrollment in public insurance and was administered before the first public enrollment period occurred under 
the new law. 

10 Adults in the Bronx were more likely to report at least two dental checkups than adults in Memphis. This 
finding is consistent with reward earnings. Fewer adults in Memphis had dental insurance coverage than those 
in the Bronx at the start of the program, which was cited as a barrier to earning the reward because of the out- 
of-pocket cost of dental care. Though advisors in Memphis emphasized connecting families to low-cost or free 
dental services in the later years of the program, the significant difference in impacts between the two cities 
persisted. See Dechausay, Miller, Quiroz-Becerra (2014). 

"Borchgrevink, Snyder, and Gehshan (2008). 

12 As of 2007, Tennessee did not offer dental coverage for adults under Medicaid; see McGinn-Shapiro 
(2008). Disparities in access to oral health by income level have also been well documented; see Fisher-Owens 
et al. (2008). 

13 As discussed in Chapter 2, program staff also connected participants with dental practitioners who pro- 
vided low-cost services or used a sliding scale for their billing. These services reduced the burden of upfront, 
out-of-pocket expenses. 
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Table 5.1 

Impacts on Parents' Receipt of Services and Health Outcomes 



Program 

Control 

Difference 

Effect 

Outcome 

Group 

Group 

(Impact) 

P- Value Size 

Health care visits in past 12 months (%) 

Has seen health professional for any reason 

92.7 

91.7 

1.0 

0.397 

Had a health checkup 

88.4 

87.6 

0.8 

0.569 

Has seen dentist for any reason 

71.2 

61.0 

10.2 *** 

0.000 

Had 1 or more dental checkups 

66.3 

54.0 

12 3 *** 

0.000 

Had 2 or more dental checkups 

36.1 

22.5 

13.6 *** 

0.000 

Respondent's health care use (%) 

Has a usual source of health care 

94.4 

91.9 

2.5 ** 

0.030 

Clinic or health center 

53.9 

53.4 

0.5 

0.836 

Doctor's office 

30.6 

27.9 

2.6 

0.181 

Hospital emergency room 

5.2 

5.8 

-0.6 

0.541 

Hospital outpatient department 

4.3 

4.6 

-0.2 

0.798 

Other 

0.4 

0.2 

0.2 

0.364 

Health insurance coverage 
in previous month (%) 

Respondent had health insurance 

92.6 

92.3 

0.3 

0.799 

Unmet health needs due to cost 
in past 12 months (%) 

Did not get needed medical care 

11.1 

11.2 

-0.1 

0.937 

Did not fill prescription 

18.5 

19.6 

-1.1 

0.544 

Health status and outcomes 

Smokes cigarettes (%) 

15.7 

18.3 

-2.6 

0.108 

Has any medical condition 1 (%) 

52.5 

54.2 

-1.7 

0.403 

Treated for any medical condition 2 (%) 

46.3 

48.5 

-2.2 

0.285 

Average Body Mass Index (BMI) b 

31.2 

30.7 

0.4 

0.167 0.061 

Overweight (%) 

30.7 

30.6 

0.1 

0.962 

Obese (%) 

48.3 

47.8 

0.5 

0.808 

Average self-rated health 
(1 = poor; 5 = excellent) 

3.2 

3.0 

0 J H=H=H= 

0.002 0.126 

Sample size (total = 2,016) 

1,025 

991 



(continued) 


74 







Table 5.1 (continued) 

SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for 
multiple observations per family. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The effect size is the difference between program and control group outcomes expressed as a 
proportion of the standard deviation of the outcomes for both groups combined. 

Participants were asked about the following conditions: asthma, allergies, arthritis, back pain, 
bone or joint problems, cancer, diabetes, depression, digestive problems, blood pressure, high 
cholesterol, lung disease, sinus infections, weight conditions, or other specified problems. The four 
most reported conditions were asthma, diabetes, high blood pressure, and high cholesterol. 

b W eight categories are from the National Institutes of Health. Underweight is defined as 
having a BMI of less than 18.5. Normal weight is defined as having a BM1 between 18.5 and 24.9. 
Overweight is defined as having a BMI between 25.0 and 29.9. Obesity is defined as having a BMI 
of at least 30. About 4.5 percent of the sample was excluded from this analysis because of missing 
data. 


Rewards for preventive visits were intended to encourage other health outcomes, such 
as connection to health care and insurance. Though program participants did not experience a 
significant increase in medical checkups, the program appears to have had a small, statistically 
significant effect on participants’ connection to a “regular source of care,” an increase of 2.5 
percentage points over the control group. It did not have a statistically significant impact on 
where program group members sought that regular source of care. It is possible that the pro- 
gram’s overall focus on health encouraged some program members to seek out more regular 
health care, however this outcome is a secondary one and cannot be considered confirmatory. A 
very high number of both program and control group members already had a regular source of 
care, even relative to the general population. 14 

Family Rewards 2.0 did not have any significant impacts on program participants’ 
health insurance coverage. Although many families in the program already had public health 
coverage, many low-income families move on and off of coverage over time. The consistency 


14 National estimates for this type of measure suggest that 17 percent of adults 18 years of age and over 
were without a usual place of health care. Of those with a usual place of care, 74 percent considered a health 
maintenance organization doctor’s office to be their usual place of health care, 22 percent considered a clinic or 
health center to be their usual place of health care, and 3 percent considered a hospital emergency room or 
outpatient department to be their usual place of health care; see Blackwell, Lucas, and Clarke (2014). Typical- 
ly, health coverage is much lower among two populations that were not eligible to enroll in the program: 
childless adults and undocumented immigrants. 
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of public coverage also varies from state to state, 15 and families in Tennessee tend to move on 
and off of coverage more frequently than those in New York. In this case, the program might 
have increased the continuity of coverage by increasing the use of more regular medical visits. 
Overall health insurance coverage proved to be extremely high for all participants, 92.6 percent 
for the program group and 92.3 percent for the control group, and the program had no effect on 
this outcome. 16 


Impacts on Health Outcomes 

Health incentives in Family Rewards 2.0 were intended to encourage participants to seek and 
strengthen their bond with a primary care provider through an annual non-emergency checkup. 
Program designers hoped that this result would improve participants’ health status through 
better diagnosis and management of chronic health conditions. Program designers also hoped 
that a general focus on health could have “spillover” effects onto other risk-taking behavior, 
such as smoking and lifestyle habits that lead to obesity, both of which contribute to negative 
health outcomes and costly health interventions. 17 

Family Rewards 2.0’s only impact on program participants’ health was on their self- 
reported health status (Table 5.1). Participants were asked to rate their health on a scale from 
one (poor) to five (excellent). This outcome was reported as an average and was 0.1 points 
higher on this scale for program participants. The effect size better indicates the practical 
significance of this difference between the program and control group; 18 effect sizes less than 


15 Medicaid (2015). 

16 Family Rewards 1.0 also collected and analyzed administrative records on Medicaid coverage for adult 
participants. Data were analyzed separately for parents receiving Temporary Assistance to Needy Families 
(TANF) or Safety Net Assistance (SNA) and indicated that a high percentage of these participants were 
receiving Medicaid coverage consistently throughout the study. While some families may be at risk for 
“churn,” or having gaps in health insurance coverage because of failure to meet administrative obligations to 
reenroll in Medicaid, individuals receiving TANF benefits are automatically enrolled. Additionally, no 
statistically significant difference between participants receiving TANF or SNA in the program and control 
groups was identified at any point during Family Rewards 1.0. Because receipt of TANF or Supplemental 
Nutrition Assistance Program (SNAP) was a condition of program enrollment in Family Rewards 2.0 and 
families were already well connected to health insurance, the research team did not analyze Medicaid data for 
Family Rewards 2.0. 

17 Fontaine (2003); Mokdad et al. (2003); Mokdad, Marks, Stroup, and Gerberding (2004); Must et al. 
(1999). 

18 The “Effect Size” column in the table indicates the effect size, or magnitude, of the difference between 
the program and control group outcomes. The research team calculated the effect size by dividing the estimate 
of the difference by the standard deviation for the outcome among the control group members. Because the 
effect size is in uniform, standard deviation units, it is possible to compare the sizes of impact estimates for 
different outcomes in this evaluation and to compare Family Rewards 2.0 impacts with the impacts of other 
programs that have been evaluated. Kazis, Anderson, and Meenan (1989). 
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0.20 are generally considered small in the behavioral sciences. 19 The effect size for parents’ 
self-rated health (0.126) is small by this standard, but indicates a difference worth noting given 
that health is difficult to influence through social interventions. 20 Despite positive impacts on 
participants’ self-reported health status, Family Rewards 2.0 did not appear to impact the 
diagnosis or treatment of chronic health conditions. It also did not have any notable impacts on 
risk-related behaviors or conditions. 


Impacts by Self-Rated Health Status at Baseline 

The effectiveness of Family Rewards 2.0 on improving health outcomes and increasing the 
receipt of health care services may be related to the initial health status of its participants. If 
those who initially suffered from poor health were in that condition because they did not utilize 
access to health care services, Family Rewards 2.0 might lead to improved health outcomes for 
this group. Additionally, participants with poor health at study entry could have more room for 
improvement in certain outcomes than those with good health. Larger effects for those in poorer 
health were found in Family Rewards 1.0. 

To examine these hypotheses, Table 5.2 presents important health outcomes by study 
participants’ self-reported health status at random assigmnent. The research team placed 
participants who rated their health as being excellent, very good, or good in one group, and 
those who rated their health as fair or poor in another group. The primary purpose of this 
analysis is to compare whether the impact for one group (those in fair or poor health) is signifi- 
cantly different from the impact for another group (those in better health). If the difference 
between these two impacts is not statistically significant, as indicated by daggers in the right- 
most column of the tables, then it is not possible to conclude with confidence that the program 
worked any differently across the two groups. 

Table 5.2 shows that Family Rewards 2.0 improved the health status of program partic- 
ipants more for those who had rated their health as poor or fair at random assignment than it did 
for those who had initially rated their health as excellent, very good, or good. The difference 
between the program’s impact on this measure for the two subgroups is statistically significant. 
The program’s impact on self-rated health was a 0.3 increase on a five-point scale for those with 
fair or poor health at random assigmnent, with the effect size indicating a notable improvement 
for this subgroup. 


19 Cohen (1988). 

20 Belfield and Levin (2007); Maciosek et al. (2006). 
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Table 5.2 


Impacts on Parents' Health Outcomes, by Self-Reported Health Status at Baseline 

Program Control Difference Effect 

Outcome Group Group (Impact) P- Value Size 

Self-rated health status at baseline 
(excellent, very good, or good) 

Average self-rated health 


(1 = poor; 5 = excellent) 

3.4 

3.3 

0.1 * 

0.092 0.086 

ft 

Smokes cigarettes (%) 

15.3 

17.3 

0.0 

0.280 


Has any medical condition 1 (%) 

43.5 

44.3 

- 0.8 

0.745 


Treated for any medical condition 11 (%) 

37.8 

38.3 

- 0.6 

0.819 


Sample size (total = 1,499) 

771 

728 




Self-rated health status at baseline 

(fair or poor) 

Average self-rated health 
(1 = poor; 5 = excellent) 

2.4 

2.1 

Q ^ H=H=H= 

0.000 0.329 

ft 

Smokes cigarettes (%) 

16.5 

21.5 

-4.9 

0.154 


Has any medical condition 1 (%) 

78.3 

83.2 

-4.9 

0.152 


Treated for any medical condition 11 (%) 

71.2 

78.6 

_7 4 ** 

0.044 


Sample size (total = 511) 

254 

257 





SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Statistical significance levels for differences in impacts across subgroups are indicated as follows: 
fff = 1 percent; f f = 5 percent; f = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The effect size is the difference between program and control group outcomes expressed as a 
proportion of the standard deviation of the outcomes for both groups combined. 

a Participants were asked about the following conditions: asthma, allergies, arthritis, back pain, 
bone or joint problems, cancer, diabetes, depression, digestive problems, blood pressure, high 
cholesterol, lung disease, sinus infections, weight conditions, or other specified problems. The four 
most reported conditions were asthma, diabetes, high blood pressure, and high cholesterol. 
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Though the research team observed no other statistically significant differences in im- 
pacts by baseline health status, one other exploratory finding should be noted. The program led 
to a statistically significant decrease in the receipt of treatment for any medical condition among 
members of the fair or poor health subgroup. Although this impact is not statistically different 
from the impact for the better health subgroup, it may be associated with the positive effect on 
self-reported health status observed for the fair or poor health subgroup and may be worthy of 
future study. 


Impacts on Children’s Health Care and Health 

As a two-generational program, Family Rewards 2.0 extended health rewards to parents and all 
children in the family. Using a similar logic to the one applied to parents, program designers 
hoped that emphasizing preventive health services would lead to an improved connection to 
health care and other health outcomes for children. On the 24-month survey, parents were asked 
to answer questions related to these outcomes for one child in their household who was entering 
ninth or tenth grade at study enrollment. Table 5.3 presents outcomes for these focal children. 
Similar to adults, high rates of enrollment in health insurance among children in both the 
program and control groups left little room for Family Rewards 2.0 to improve connection to 
care and preventive care practices for focal children in the program group. Family Rewards 2.0 
did not produce statistically significant changes in focal children’s non-emergency preventive 
health checkups. 

The program did lead to an increase in biannual visits to the dentist among children in 
the year before the survey. The magnitude of the impact for a second dental checkup (16 
percentage points) was much larger than just one checkup (3.8 percentage points). This effect 
implies that the dental reward was effective at helping children to meet the American Dental 
Association recommendation of maintaining more regular checkups. 21 Access to dental care 
also has the potential to help children avoid more serious medical conditions later in life. 22 

Across both the program and control groups, adults were much less likely to see the 
dentist, even once, compared with their children. The impact on seeing a dentist at least once 
was likely much higher for adults than their children because of this difference. Reported rates 
of reward receipt are consistent with this finding. Family Rewards 2.0 did not produce statisti- 
cally significant changes in focal children’s source of usual care or any other health outcomes. 
Without an effect on the primary outcome of preventive health checkups, it is unsurprising that 
the program did not show effects on health status or source of care. 


2 'American Dental Association (2014). 
22 Glied and Neidell (2010). 
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Table 5.3 

Impacts on Focal Child's Receipt of Services and Health Outcomes 



Program Control Difference 

Effect 

Outcome 

Group 

Group 

(Impact) 

P- Value Size 

Health care visits in past 12 months (%) 





Has seen health professional for any reason 

97.2 

96.6 

0.5 

0.505 

Has seen personal pediatrician for any reason 

74.0 

74.0 

0.0 

0.983 

Had health checkup or got shots 

94.9 

94.4 

0.5 

0.614 

Has seen dentist for any reason 

92.8 

89.3 

3.5 

*** 0.010 

Had 1 or more dental checkups 

92.3 

88.5 

3.8 

*** 0.005 

Had 2 or more dental checkups 

62.4 

46.4 

16.0 

*** 0.000 

Respondent's health care use (%) 





Has usual source of care when sick 

94.8 

95.2 

-0.4 

0.706 

Clinic or health center 

55.4 

56.0 

-0.5 

0.821 

Doctor's office 

32.1 

31.5 

0.6 

0.782 

Hospital outpatient department 

3.1 

3.8 

-0.7 

0.421 

Hospital emergency room 

3.6 

3.6 

0.1 

0.936 

Other 

0.5 

0.3 

0.2 

0.570 

Health insurance coverage in previous month (%) 




All dependent children had health insurance 3 

95.4 

94.7 

0.7 

0.468 

Health outcomes 





Child's health (1 = poor; 5 = excellent) 

4.0 

3.9 

0.0 

0.293 0.047 

Child has any health condition 1 ’ (%) 

20.8 

20.4 

0.5 

0.788 

Sample size (total = 1,881) 

976 

905 




(continued) 


Conclusion 

Similar to those in the original program, Family Rewards 2.0’s health incentives were designed 
to connect low-income families to preventive health care resources. A high proportion of 
families were already covered by health insurance and were practicing preventive health care, 
which limited the program’s ability to further improve some health care practices and connec- 
tion to care. Despite this limitation, the program did produce a small number of impacts. The 
program increased dental visits for both adults and children. It also increased adult participants’ 
self-rated health status, particularly for adults who reported being in poor health at study entry. 
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Table 5.3 (continued) 


SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: This table presents outcomes only for focal children who were living in the household at 
the time of the survey interview. 

Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for 
multiple observations per family. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

The effect size is the difference between program and control group outcomes expressed as a 
proportion of the standard deviation of the outcomes for both groups combined. 

“Child-related health insurance measures were calculated for sample members with at least 
one child at the time of the survey. 

b Participants were asked about the following conditions: learning disability, attention deficit 
disorder or attention deficit hyperactivity disorder, autism, depression or other emotional 
condition, speech disorder or delay, asthma, heart problems, chrinic illness, weight conditions, or 
other specified problem. 


Family Rewards 1.0 produced similar impacts on dental visits and participants’ self- 
rated health status. It also had some impacts that were not observed in Family Rewards 2.0, 
including small increases in continuous health coverage and the treatment or diagnoses of 
certain conditions for adults and children. It is possible that these additional impacts were not 
replicated in Family Rewards 2.0 because participants were already highly connected to health 
care. Comparing the two programs’ impacts on all of these measures, it seems likely that the 
rewards for preventive health visits, such as dental checkups, can help comiect participants to 
better health care resources. 
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Chapter 6 

Impacts on Parents’ Education, Training, and Work 


The work incentives in Family Rewards 2.0 were part of a package of rewards intended to 
encourage behavioral changes in the areas of education, health, and work. The program design- 
ers expected the incentives structure to affect a family’s decisions about employment in a 
variety of ways. If the reward for employment was large enough, parents might decide to enter 
employment or increase their work hours, since the reward payments could serve as a de facto 
boost to wages. On the other hand, since the program allowed families to earn rewards through 
a range of activities, effects on work and training might depend on how families decide to 
complete health, education, and work-related activities to earn rewards. Parents and children 
could focus on some activities to the exclusion of others. Similarly, additional money that 
families received from the education and health rewards might reduce the need to work and 
offset some or all of the work incentive that these rewards offer. 

This chapter examines whether Family Rewards 2.0 prompted parents to pursue any 
further education or training and whether it affected their employment behavior during the three 
years the rewards were offered. It also discusses how the program’s effects compare with those 
found for Family Rewards 1.0. In sum, the program increased participation in training but did 
not result in higher rates of degree or certificate receipt. It also produced small negative effects 
on parents’ employment and earnings. 


The Work and Training Rewards 

Family Rewards 2.0 provided opportunities for parents to earn rewards for working or for 
earning a General Education Development (GED) certificate. Parents could earn $150 a month 
for working full time (30 or more hours per week) by providing evidence each month that they 
had worked at least 120 hours for the month. Additionally, parents could earn a one-time $400 
reward for earning a GED certificate during the program period. These rewards were slightly 
different from those in Family Rewards 1.0. In that program, parents earned $300 every two 
months for averaging 30 or more hours of work per week, 1 and they could earn rewards for 
taking training classes, in addition to pursuing and earning a GED certificate. 


The Family Rewards 2.0 payment structure for work rewards made it easier for individuals with less 
stable employment to earn rewards for full-time work, since the rewards were paid on a monthly rather than 
a bimonthly schedule. With the bimonthly schedule used in Family Rewards 1.0, parents who worked full 
time for one month but not the other in the same payment period would not have earned a work reward for 
that period. 
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As shown in Table 2.1, families earned an average of $2,926 in work-related rewards 
while enrolled in the program. Just over half of the program group families earned at least one 
reward payment for maintaining full-time work, and 2 percent received a reward for earning a 
GED certificate. 


Summary of Impacts on Work and Training 

The research team examined the effects of the program using data from the 24-month survey 
and administrative data from each state’s unemployment insurance (UI) wage records. The UI 
data cover the full study sample for 3 years after random assignment. The survey was fielded 
about 2 years after random assignment, before some of the Year 3 changes were implemented. 
Some changes in advisements — in particular, the full implementation of motivational inter- 
viewing — began in Year 3, which ran from September 2013 through August 2014. During 
Year 3, advisors also more actively focused on program participants who had not received 
work-related rewards in the past. 

The primary outcomes examined in this chapter are educational attainment, employ- 
ment rates, and earnings. In sum, the program did not boost educational attainment and pro- 
duced a reduction in employment and earnings. Two years after random assignment, program 
group members were somewhat more likely to have a trade license or training certificate than 
control group members, but they were no more likely to have a degree. The program produced 
zero to negative impacts on employment and earnings, depending on the data source. The 
reduction in earnings from UI data was particularly large for program group members in 
Memphis, as Chapter 7 discusses, although the negative effects seemed to have diminished 
during the second half of Year 3. 


Impacts on Parents’ Education and Training 

This section presents the program’s effects on parents’ participation in education and training 
activities and on certificate and degree receipt. Education and training data for parents were 
only available on the survey, which asked respondents about current education levels as well as 
participation in education and training activities since random assignment. 

Table 6.1 shows that at the 24-month point, about 32 percent of the control group re- 
ported participating in some education, training, or employment program activity since the time 
of random assignment, 43 percent reported having a trade license or training certificate, and 78 
percent reported having any high school diploma or high school equivalency credential, trade 
license, training certificate, or college degree. Within 24 months of study entry, Family Re- 
wards 2.0 increased participation in education and training programs by 4 percentage points and 
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Table 6.1 


Impacts on Educational Attainment and 
Participation in Educational and Employment Activities 


Outcome (%) 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Has any degree, license, or certificate 

78.5 

77.8 

0.7 

0.651 

Has any trade license or training certification 

46.6 

42.7 

3.9 * 

0.073 

Home health aide 

10.1 

11.6 

-1.5 

0.223 

Nurse's aide/nurse's assistant 

5.2 

3.9 

1.4 

0.145 

Child care/teaching 

4.7 

3.6 

1.1 

0.226 

Medical assistant 

5.0 

2.9 

2.1 ** 

0.015 

Security 

1.8 

1.2 

0.6 

0.299 

Other 

20.0 

19.2 

0.8 

0.668 

Highest degree or diploma 

High school equivalency certificate 

15.3 

15.6 

-0.3 

0.840 

High school diploma 

33.8 

35.2 

-1.3 

0.486 

Associate's degree 

10.8 

9.1 

1.7 

0.189 

Bachelor's degree or higher 

5.7 

6.4 

-0.7 

0.481 

Ever participated in any education, training, 

or employment activity 

36.3 

32.2 

4 [ ** 

0.049 

Adult basic education, high school equivalency, 

8.5 

8.7 

- 0.2 

0.870 

or high school classes 

English as a second language classes 

7.7 

6.1 

1.6 

0.139 

College courses for credit 

13.6 

13.1 

0.5 

0.759 

Vocational training 

10.3 

9.5 

0.8 

0.562 

Other educational, training, or employment 

program activities 

5.9 

4.6 

1.3 

0.183 

Currently participating in any education, training. 

or employment activity 

12.5 

9.6 

29 ** 

0.039 

Sample size (total = 2,016) 

1,025 

991 




SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Rounding may cause discrepancies in calculating sums and differences. 

Percentages may sum to more than the overall participation rate because sample members could list 
more than one response. 
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certificate receipt by 4 percentage points. It is possible that the reward payments families 
received from the other domains allowed parents to pay for training classes if they were seeking 
some type of certification for future employment. However, the program had no statistically 
significant effect on further educational attainment — including no effect on earning a GED 
certificate, which would have merited a reward payment. 

An increase in training activities could have affected how much parents were able to 
work in the short tenn while participating in their training classes. The next section considers 
whether these effects on training were associated with effects on employment over the three- 
year follow-up period. 


Impacts on Employment and Earnings 

The research team calculated employment impacts from both UI and survey data. UI earnings 
were available quarterly for each sample member in the study, so the follow-up for most sample 
members covered July 201 1 through March 2015. The survey asked about employment at the 
time of the survey and in the year before the survey was conducted for all survey respondents. 
The survey also provided rich information about respondents’ job characteristics, such as hourly 
wages, and employer-provided benefits — information that was not available on UI wage 
records. Typically, the survey data captured more employment than UI records because some 
types of jobs, including “off-the-books” jobs, federal government jobs, out-of-state jobs, and 
self-employment, are not reported to the state UI systems. 

Table 6.2 shows that about 60 percent of the control group was employed in Year 3, 
with just over half of them employed in each quarter. Family Rewards 2.0 reduced work in UI- 
covered jobs by about 4 percentage points over three years and reduced earnings from work by 
about $2,000. This amount translates to about 7 percent of the earnings over three years for the 
average control group member. 

Economists often refer to a reduction in work (to possibly spend more time in nonpaid 
activities) in response to an increase in household income as an “income effect.” Such an 
income effect was observed in Family Rewards 2.0 for the full study sample and not concen- 
trated in a specific subgroup, as Chapter 7 discusses. Quarterly impacts on employment rates 
and earnings amounts indicate that the negative effects on employment and earnings persisted 
only through the middle of Year 3; the negative impacts were no longer statistically significant 
in the last two quarters of the year (not shown). These effects may have been due to implemen- 
tation changes in Year 3 that focused on conducting motivational interviews with participants 
who were not earning the full-time work reward payments. It is also possible that families were 
increasing their work activities to prepare for the end of the Family Rewards 2.0 program, when 
the reward payments would stop. 
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Table 6.2 


Impacts on Unemployement Insurance-Covered 
Employment and Earnings, Years 1 to 3 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Ever employed (%) 

Year 1 

58.6 

61.1 

-2.5 * 

0.057 

Year 2 

58.0 

60.9 

-2.9 * 

0.053 

Year 3 

58.5 

62.7 

_4 2 *** 

0.007 

Years 1 to 3 

68.0 

71.9 

g *** 

0.005 

Average quarterly employment (%) 

Year 1 

48.8 

51.5 

_2 7 ** 

0.012 

Year 2 

49.5 

51.9 

-2.4 * 

0.065 

Year 3 

50.6 

53.2 

-2.6 * 

0.071 

Years 1 to 3 

49.6 

52.2 

-2.6 ** 

0.018 

Total earnings ($) 

Year 1 

8,404 

9,054 

-650 *** 

0.007 

Year 2 

9,124 

10,044 

_92i *** 

0.006 

Year 3 

10,156 

10,619 

-463 

0.244 

Years 1 to 3 

27,684 

29,718 

-2,034 ** 

0.019 

Sample size (total = 2,565) 

1,286 

1,279 




SOURCES: MDRC calculations using data from New York State unemployment insurance 
(UI) wage records and Tennessee Department of Labor and Workforce Development UI 
wage records. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre- 
random assignment characteristics of sample members. Standard errors were adjusted to 
account for multiple observations per family. Statistical significance levels are indicated as 
follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Dollar averages include zero values for sample members who were not employed. 

This tables includes only employment and earnings in jobs covered by the New York 
State and Tennessee UI programs. It does not include employment outside of either state, or 
in jobs not covered by the UI system (for example, "off-the-books" jobs and federal 
government jobs). 
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Table 6.3 presents employment data as reported on the 24-month survey. The program 
had no effect on survey-reported employment, either at the time of the survey or during the 
previous year. According to the survey, about two-thirds of parents were employed at some 
time in the 12 months before they responded to the survey. 

Table 6.3 also shows some characteristics of the jobs that sample members held. Em- 
ployed control group members were earning, on average, $11 an hour in their current jobs. 
About 43 percent of the control group, or three-quarters of those who were employed, were 
working full time at the time of the survey. Almost 40 percent of the control group, or two- 
thirds of those who said they were employed, had a regular daytime shift. A small proportion of 
the control group was self-employed or working for a “temp” agency at the time of the survey. 
The program did not have effects on any of these job characteristics, even though a larger 
proportion of parents in the program group reported recently looking for work. It is possible that 
the program increased job search efforts that did not result in actual employment. 


Subgroup Impacts on Parents’ Education, Training, 
and Employment 

This section presents subgroup impacts on education, training, and employment defined by 
parents’ educational level at the time of study entry. Chapter 7 discusses impacts for additional 
subgroups. In Family Rewards 1.0, families who were less disadvantaged — as defined by 
those in which parents had at least a high school diploma or equivalency certificate, or were 
employed when they enrolled in the study — experienced a small increase in educational 
attainment but no effect on earnings. 

Family Rewards 2.0 had similar impacts on training. The program’s positive impacts on 
certificate receipt were concentrated among the parents who were more educated when they 
enrolled in the study, as shown in Table 6.4. 2 It is possible that parents who had not yet passed 
their GED exams before 2014 felt less inclined to take advantage of the GED reward offer 
because of the changes to the GED exam in 2014 that made it more difficult. 3 About 60 percent 
of the survey respondents who were interviewed in 2014 did not have a high school diploma or 
equivalency certificate when they enrolled in the study. 


2 The difference of the impact on trade license and certificate receipt across parents with differing educa- 
tion levels was statistically significant. The difference of the impact on participation in training was not 
statistically significant, but the pattern of impacts is similar — the program effects were larger for parents who 
were more educated and smaller for parents without a high school diploma or equivalency certificate at the 
time of random assignment. 

'Porter (2015). 



Table 6.3 


Impacts on Employment and Job Characteristics 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Employment status (%) 





Employed at the time of the survey 

55.9 

57.1 

-1.2 

0.528 

Employed in past year 

66.4 

67.4 

-1.0 

0.586 

Characteristics of current iob a 





Average hourly wage ($) 

11.31 

11.39 



Less than $7.00 (%) 

5.6 

7.0 

-1.4 

0.195 

$7.00 to $8.99 (%) 

9.8 

9.4 

0.5 

0.722 

$9.00 or more (%) 

32.6 

30.4 

2.1 

0.266 

Not reported (%) 

7.9 

10.3 

-2.4 * 

0.055 

Worked at least 30 hours per week (%) 

41.9 

43.2 

-1.3 

0.520 

Average weekly earnings ($) 

193 

199 

-6 

0.554 

Usual work schedule (%) 





Regular daytime shift 

37.3 

39.2 

-2.0 

0.316 

Regular evening/night shift 

6.2 

7.7 

-1.5 

0.176 

Self-employed (%) 

9.5 

8.3 

1.3 

0.301 

Worked for a temporary employment agency (%) 

6.0 

6.1 

-0.1 

0.910 

Received any employer-provided benefit 15 (%) 

37.7 

38.7 

-1.0 

0.564 

Employment search (%) 





Looked for work in the previous 4 weeks 

34.2 

30.6 

3.5 * 

0.081 

Sample size (total = 2,016) 

1,025 

991 




(continued) 


However, the program’s effects on earnings were somewhat different. Family Rewards 
1.0 reduced earnings only among parents without a high school diploma or equivalency certifi- 
cate. Family Rewards 2.0, in contrast, reduced earnings among the full program group sample. 
A pattern of statistically significant and negative effects on earnings and employment were 
consistently larger for parents who were more educated at the time of random assignment, 
although the impact differences across the less educated and more educated subgroups were not 
statistically significant. While it is possible that some parents cut back on work hours to partici- 
pate in training, as some program participants indicated in field interviews, a separate calcula- 
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Table 6.3 (continued) 

SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random 
assignment characteristics of sample members. Standard errors were adjusted to account for multiple 
observations per family. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

Italic type indicates comparisons that are nonexperimental. Statistical tests were not performed. 
Rounding may cause slight discrepancies in calculating sums and differences. 

a lf a respondent worked multiple jobs at the time of the interview, then only the characteristics of the 
primary job are reported. (The job at which the respondent worked the most hours is considered 
primary.) Respondents who were not employed at the time of the survey are included in all the current 
job characteristics measures, except for average hourly wage. The average hourly wage measure 
includes only respondents who were employed at the time of the survey. 

b This category includes benefits that are or eventually will be offered, regardless of whether the 
respondent received them. Sample members were asked whether they received paid sick days, paid 
vacation days, paid holidays, dental benefits, retirement benefits, or health or medical insurance from 
their current employers. 


tion of these subgroup impacts by city showed that the subgroups who experienced positive 
training effects were not the ones who experienced negative employment effects (not shown). 4 

The research team also estimated impacts by city (discussed more fully in Chapter 7) to 
assess whether the reductions in work occurred in each of the different labor markets. The 
findings show that reductions in Ul-covered work for three years after the time of random 
assignment were more pronounced in Memphis, while reductions in work reported at the time 
of the survey were more pronounced in the Bronx. The reason for the differences across cities is 
not clear. 

The reduction in work and earnings was likely a response to the substantial rewards that 
these families were earning from the program’s health and education rewards. Since families 
were receiving reward payments, they could reduce work hours and, in some cases, stop 
working entirely, without lowering the amount of financial resources they had before the 


4 The analysis showed that in the Bronx the program increased training for those with less education at 
baseline without affecting their work behavior. Families with more educated adults cut back on work hours and 
did not participate in more training. In Memphis, the education subgroups showed no statistically significant 
movement in either training or work. While the sample sizes in this separate investigation were too small to 
draw conclusive findings, and differences in impacts across these subgroups were not statistically significant, 
the analysis does show that sample members were generally not leaving work to pursue additional training. 
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Table 6.4 


Impacts on Education, Training, and Employment Activities, 
by Respondent's Education Level at the Time of Random Assignment 


No High School Diploma or High School Diploma or 

Equivalency Certificate Equivalency Certificate 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 


P-Value 

Survev responses (%) 










Has any degree, license, or certificate 

49.2 

49.8 

-0.7 

0.855 

97.0 

95.5 

1.6 


0.142 

Has any trade license or 
training certification 

37.7 

39.1 

-1.4 

0.674 

52.5 

45.1 

7.5 

sjssjssj: 

0.009 ff 

Ever participated in any education, 
training, or employment activity 

32.2 

29.3 

2.9 

0.377 

39.1 

33.7 

5.4 

** 

0.047 

Employed at the time of the survey 

48.8 

50.5 

-1.8 

0.542 

60.4 

60.7 

-0.3 


0.909 

Worked at least 30 hours per week 

36.0 

36.7 

-0.8 

0.805 

45.9 

46.8 

-0.9 


0.735 

Sample size (total = 769) 

396 

373 



618 

609 




Unemployment insurance records 

Employment in Years 1 to 3 (%) 

59.5 

62.3 

-2.8 

0.222 

73.7 

78.0 

-4.3 

** 

0.014 

Total earnings in Years 1 to 3 ($) 

20,735 

21,715 

-980 

0.318 

32,287 

34,941 

-2,655 

** 

0.040 

Sample size (total = 1,005) 

510 

495 



760 

770 





SOURCE: MDRC calculations using data from the Family Rewards 24-month survey, New York State unemployment insurance (UI) wage 
records, and Tennessee Department of Labor and Workforce Development UI wage records. 

NOTES: Sample sizes vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. 
Standard errors were adjusted to account for multiple observations per family. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Rounding may cause discrepancies in calculating sums and differences. 

Statistical significance levels across subgroups are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 


program started. Although it is possible that some parents decided to forego immediate em- 
ployment opportunities to pursue education and training, very few parents who were not 
working when they responded to the survey listed education or training as a primary reason for 
not working, instead citing either their own or a family member’s health issues. 


Barriers to Finding and Maintaining Full-Time Work 

About 40 percent of study participants were not working at the time of the survey, and about 60 
percent were not working at least full time. Table 6.5 presents reasons that survey respondents 
in the program group reported for either not working full time or not working at all. Although 
the reasons do not explain why program group members cut back more on work than did the 
control group members, they provide some context as to why parents could not work as much 
as they as they might have wanted. Among program group members who were working part 
time at the time of the survey, about a third of them were unable to increase their hours or find a 
full-time job. 

In field interviews, participants expressed difficulty in obtaining the minimum 120 
hours per month of work that were required to earn the full-time work reward payment. One 
participant described the instability of her hours as a home health aide and how difficult it was 
for her to accrue enough hours to earn the reward: 

[T]his year has been very bad for me, very bad, because 1 missed three months’ 
worth of $150.00. Three months. And they called me, “What happened? You 
know, you’re not sending in your pay stubs”. . . I said, “I didn’t work a week, and 
then I had to wait another week, and then I only had like 1 8 hours.” I mean, it 
was really frustrating. But as home care attendants, if you’re not working, you’re 
not getting paid, you don’t have any hours. So it messes you up. 

Program advisors, despite encouraging families in the program to increase their work 
hours so that they could earn the full-time work reward, indicated that they felt limited in 
providing concrete guidance to individuals on how to do so: 

We have ... so many people who come so close. You know, they’re in between 
100 and 119 ... so they’re working, you know, they’re working part time, but 
they’re working consistently, but they’re not meeting the 120 ... It’s hard for 
people to hit the 120. We talked to people about ... talking to your employer 
about getting more hours, or we’ve had some people getting second jobs.” 

About 12 percent of program group respondents working part-time reported that their 
caretaking responsibilities at home limited their ability to work full time. Interviews with 
participants reflected this reality — one participant had to spend a lot of time taking care of her 
sick mother; another participant had to care for a grandmother whose leg was amputated. 
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Table 6.5 

Respondents Not Working Full Time 

Program 

Outcome (%) Group 


Among respondents working part time at the time of the survey 

Main reason for not working full time 

Inability to a find full time job or work more hours 3 1 .2 

Need to take care of home or family 11.6 

Attendance at school or training 4.4 

Respondent's illness or disability 4.4 

Someone else's illness or disability 2.9 

Inability to find adequate child care 2.9 

Pregnancy 0.7 

Other 35.5 

Sample size 138 


Among respondents who were not working at the time of the survey 

Main reason for not working 

Respondent's illness or disability 40.8 

Inability to find work 18.7 

Need to take care of home or family 8.6 

Someone else's illness or disability 3.5 

Being temporarily laid off or on temporary leave 2.8 

Inability to find adequate child care 4.2 

Attendance at school or training 3.3 

A criminal record that makes finding work hard 0.9 

Pregnancy 0.7 

Retirement or aging 0.2 

Other 16.3 

Average wage that respondents would consider before taking a job offer 

Less than $7.00 4.5 

$7.00 to $8.99 10.0 

$9.00 or more 44.7 

Not reported 40.8 

Sample size 441 

SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: Sample sizes may vary because of missing values. Statistical tests were not 

performed. 
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Program group respondents who were not employed at the time of the survey most fre- 
quently reported that the reason for not working was the respondents’ illness or disability; about 
41 percent of nonworking respondents reported this reason as the main one. About one-fifth of 
nonworking respondents also reported that they were unable to find work. 


Conclusion 

Through its first two years, Family Rewards 2.0 increased the proportion of parents who earned 
a trade license or training certificate and the proportion of parents who participated in an 
education or training activity. The education and training impacts were concentrated primarily 
among adults who already had a high school diploma or equivalency certificate when they 
enrolled in the study. The program reduced Ul-covered work and did not produce statistically 
significant impacts on survey-reported work. The average program group adult earned about 
$2,000 less from working than the average control group member over three years, which is 
about one-third of what each family earned in reward payments, on average, over the full 
program period. The reduction in work may have had additional implications for families, 
including a possible reduction in Earned Income Tax Credit benefits. 

These findings are similar to those in Family Rewards 1.0, which produced a small in- 
crease in college course participation and a small increase in getting a trade license or training 
certification. Family Rewards 1.0 also increased employment, mostly in jobs that were not 
reported to the UI system. However, it led to reductions in Ul-covered work for the less “work- 
ready,” 5 with no or small positive effects on survey-reported employment. 6 

Even though the reductions in work among families enrolled in Family Rewards 2.0 
were not completely unexpected based on findings from the earlier program, it is not immedi- 
ately clear why this replication program would have produced a broader income effect than did 
Family Rewards 1.0, given the smaller amount of other rewards that families earned. However, 
all sample members in the Family Rewards 2.0 evaluation had to be receiving food stamps or 
Temporary Assistance for Needy Families to be eligible, which was not a requirement for 
Family Rewards 1.0. Thus, the broader income effect on a more disadvantaged population 
makes sense in this context. The finding, however, is a concern, particularly since employment 
and training incentives have been shown to work in other places. Earlier studies have found that 


5 The research team defined the subgroups that were less “work-ready’’ by (1) parents who did not have a 
high school diploma or equivalency certificate when they enrolled in the program, and (2) parents who were 
not working when they enrolled in the program. 

6 Riccio et al. (2013). 
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providing work and training incentives in the absence of the other incentives can increase work 
and, in some cases, training. 7 

A reduction in work, particularly among more disadvantaged families, has now been 
observed in Family Rewards 1.0 and replicated in Family Rewards 2.0. This finding suggests 
that work and training incentives may be more effective in employment-focused programs 
rather than packaged with other incentives that target family or child well-being. It also serves 
as a general caution for conditional cash transfer (CCT) programs with generous incentives 
structures that may improve education and health outcomes for participants who earn the 
incentives — although there is little evidence that CCT programs in developing countries 
reduce work. 8 Program designers and administrators interested in developing and structuring 
large-scale CCT programs should weigh the benefits of education and health improvements 
against the decisions that low-income families might make about work. Without the work 
incentives, however, the program might have produced even larger negative employment 
effects. 


7 Riccio et al. (2008); Martinson and Hendra (2006); Michalopoulos et al. (2002). 

8 Fiszbein and Schady (2009). 
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Chapter 7 

Impacts by City and for Subgroups 


Earlier chapters of this report provided some evidence that Family Rewards 2.0’s effects vary 
across different types of families. The program’s effects on health outcomes, for example, 
varied by participants’ health status at study entry. It is reasonable to expect that a program such 
as Family Rewards 2.0 might have varying effects for different types of families, particularly as 
defined by their level of disadvantage or ability to meet the milestones to earn the rewards. 
Varying effects were also found to some extent in Family Rewards 1.0. 

This chapter presents Family Rewards 2.0’s effects for several subgroups of families, 
those defined by family income, parents’ education level, and parent’s work status at study 
entry. Although the earlier chapters present effects in certain domains for a few of these groups, 
this chapter presents effects across outcomes in all domains, to provide a fuller picture of the 
program’s effects. 

The chapter also presents effects by city. The report has presented overall effects for 
both cities combined, in order to provide a more generalizable assessment of the program’s 
effects. However, effects might have varied by city because of differences in the types of 
participants across the two areas and differences in the context. The implementation findings 
also suggest that the program’s effects might have varied by city. Transportation, for example, 
posed a challenge for Memphis participants in meeting with their advisors. In addition, families 
earned on average more from the rewards in the Bronx than in Memphis, indicating that the 
program may have led to bigger changes in behavior for the former group. 

In sum, the findings show that the program had fairly similar effects across the two cit- 
ies. In both cities, the program reduced poverty, improved health status, and led to more dental 
visits. Also in both cities, the program had little overall effect on children’s progress in school 
and led to some reductions in parents’ work. The results also suggest that the program had 
broadly consistent effects across more versus less disadvantaged families, with a few excep- 
tions. The program seemed to work the least well for families without an employed parent, with 
those families earning the least from the program and seeing relatively large reductions in work 
and earnings during the follow-up period. In addition, there is some suggestion that the program 
increased graduation rates among students in the study’s more advantaged families, or those 
whose families had higher incomes or whose parents were working at study entry. 
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Defining Subgroups and Estimating Impacts 

The research team specified the subgroups — defined by city, family income, parents’ educa- 
tional level, and parents’ work status — before conducting the analysis. The team selected them 
based on the understanding that measures of well-being vary greatly across groups defined by 
socioeconomic characteristics and that groups defined by those characteristics might have 
differential responses to the program’s cash incentives. In other words, different capacities or 
starting points might affect how individuals respond to the incentives. For example, the more 
“advantaged” participants — those with higher income or those who were employed or who 
had higher levels of education at study entry — may have been more likely to earn rewards and 
thus experience greater improvements in income, poverty, and material well-being. Alternative- 
ly, these same participants might have been more likely to have already met the conditions 
necessary to earn rewards, so that they earned rewards with little change in behavior. Descrip- 
tive analyses presented in the implementation report show that families who earned more in 
rewards differed substantially from those who earned much less. Among other distinguishing 
characteristics, the top earning families were more likely to have been employed at baseline and 
to have held a high school diploma or equivalency certificate. This chapter examines whether 
the program had bigger effects on their education, health, and work outcomes. 

The research team conducted the analysis by estimating impacts separately for each 
subgroup and the then assessing the differences in effects. In general, impacts were expected to 
vary to some extent across subgroups, simply as a result of natural variation around the average 
impact for the full sample. This chapter assesses whether that variation in impacts across 
subgroups is statistically significant, or beyond what would be expected to occur naturally. For 
that reason, the focus is not on whether a given impact for, say, the less-educated subgroup is 
statistically significant, but whether the difference between that impact and the impact for the 
more educated subgroup is statistically significant (indicated by daggers in the rightmost 
column of the tables). If the difference between these two impacts is not statistically significant, 
the results suggest that the effects observed for the full sample generally held across both groups 
being compared. However, differences that are notable but not statistically significant may be 
highlighted in the text that follows as exploratory and worthy of further analysis. Finally, 
because subgroup samples tended to be smaller and because the test for the difference between 
two impacts required more statistical power than a test of one impact, it is important to keep in 
mind that differences in impacts between any two subgroups needed to be fairly large in order 
to be found statistically significant. 
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Impacts by City 

Table 7.1 presents effects separately for the Bronx and Memphis. As shown in the bottom 
panel, families in Memphis earned on average $5,765 from the program over the full period, 
while families in the Bronx earned an average of $6,719. The reason for this difference is due 
largely to lower education rewards earned among Memphis families. 

Overall, as indicated by the daggers, the main differences in effects are in the area of 
work and training. As mentioned briefly in Chapter 6, the program led to a reduction in em- 
ployment reported on the survey in the Bronx, with no effects on this measure in Memphis. In 
contrast, the program led to a reduction in employment and earnings covered by unemployment 
insurance (UI) in Memphis, with less of an effect in the Bronx. The reason for this difference is 
not clear. The program’s positive effects on training, found for the full sample, were also driven 
entirely by positive effects in the Bronx. 

The remainder of the table suggests that the program’s effects in the other domains 
were fairly consistent across cities, although there are differences in the size of the effects. For 
example, the program led to a smaller reduction in poverty in Memphis than in the Bronx, 
which is probably related to the lower rewards earned on average for families in Memphis. The 
program reduced severe poverty (or the percentage of families below 50 percent of the federal 
poverty line) by similar amounts in both cities (not shown). 


Impacts by Poverty Level 

Table 7.2 presents effects separately for families who entered the study with income below 50 
percent of the federal poverty line versus families with income above that threshold. Lower- 
income families earned substantially less in rewards ($5,328 on average) than higher-income 
families ($7,103 on average), due to lower reward payments in education and work. Nonethe- 
less, the program’s effects on poverty reduction were roughly the same for both types of 
families. 

Even though lower-income families earned less in rewards, the program might nonethe- 
less have had larger effects on their behavior. For these families, the rewards represented a 
larger potential fraction of their current income and thus may have been more salient and 
powerful incentives. The table also presents effects on education, health, and work. Overall, 
there are few significant differences, with the exception of the health domain. The program 
increased health coverage for lower-income families, but had smaller effects for this group on 
two or more dental visits. Finally, there is a difference in impacts on graduation rates, with a 4.3 
percentage point increase in graduation rates for students in the higher income group. 
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Table 7.1 


Summary Program Impacts, by City 





Bronx 



Memphis 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Income, poverty, and well-being (%) 

Household income at or below the federal poverty 
level (including rewards) ab 

71.0 

78.8 

g *** 

0.005 

76.1 

77.9 

-1.8 

0.492 

Any savings 

18.3 

11.6 


0.003 

22.8 

12.5 

10.3 *** 

0.000 

Any housing/utilities material hardship in 
the past 12 months 

59.7 

60.1 

-0.5 

0.88 

69.1 

67.9 

1.2 

0.678 

Did not have enough money to buy food 
sometime in the past 12 months 

45.0 

50.2 

-5.2 

0.101 

41.8 

44.3 

-2.5 

0.417 

Very or pretty happy c 

72.3 

67.1 

5.2 * 

0.075 

80.2 

77.3 

2.9 

0.261 

Education 

Number of state core exams passed, Years 1 to 3 d,e 

3.7 

3.7 

0.0 

0.867 

1.8 

1.7 

0.1 

0.106 

Attendance rate is 95% or higher. Year 3 f (%) 

25.6 

26.8 

-1.2 

0.630 

32.7 

30.6 

2.1 

0.412 

Graduated on time (%) 

61.4 

57.2 

4.2 

0.122 

66.4 

67.5 

-1.1 

0.660 

Health 

All dependent children had health insurance 8 (%) 

94.3 

94.6 

-0.3 

0.823 

96.5 

94.8 

1.7 

0.190 

Parents' average self-rated health 
(1 = poor; 5 = excellent) 

3.1 

3.0 

0.1 

0.154 

3.2 

3.0 

q 2 *** 

0.003 

Focal child had 2 or more dental checkups in past 

12 months (%) 

60.5 

42.7 

17 g *** 

0.000 

64.3 

50.0 

14 3 *** 

0.000 
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Table 7.1 (continued) 





Bronx 



Memphis 



Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 


Work 

Ever participated in any education, training, 
or employment activity (%) 

40.0 

32.3 

7 7 ** 

0.011 

32.5 

32.2 

0.3 

0.920 

f 

Currently employed at the time of the survey (%) 

58.1 

63.0 

_4 9 ** 

0.048 

53.8 

51.3 

2.5 

0.360 

tf 

Unemployment insurance-covered employment 
earnings, Years 1 to 3 ($) 

31,270 

32,427 

-1,157 

0.354 

23,933 

26,893 

-2,960 ** 

0.013 


Reward participation. Years 1 to 4 ($) 

Among families who earned at least one reward 
Average reward amount earned h 

6,719 




5,765 





Average reward amount earned, by domain 

Education 

3,344 




2,554 





Health 

1,728 

— 

— 

— 

1,885 

— 

— 

— 


Work 

3,022 

— 

— 

— 

2,818 

— 

— 

— 


Sample size (total = 2,016) 

516 

491 



509 

500 
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Table 7.1 (continued) 


SOURCES: MDRC calculations using data from the Family Rewards 24-month survey, Children's Aid Society's Family Rewards program data. New York 
City Department of Education and Shelby County Schools administrative records. New York State unemployment insurance (UI) wage records, and 
Tennessee Department of Labor and Workforce Development UI wage records. 

NOTES: This table presents survey outcomes only for focal children who were living in the household at the time of the interview and at random 
assignment. 

Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Standard 
errors were adjusted to account for multiple observations per family. The p-value indicates the likelihood that the difference between the program and control 
groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance levels across subgroups are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Education measures include all students who were enrolled in ninth or tenth grade at baseline (2,676 sample members). Employment measures calculated 
from unemployment administrative records include all enrolled parents (2,565 sample members). 

“Monthly household income amounts equal to or greater than $10,000 were excluded from this calculation. About 4.9 percent of the sample was excluded 
from the income measures because respondents did not know or refused to provide the information. An additional 0.2 percent of the sample was excluded 
because the income provided was over $10,000. 

h Annual household income was calculated by multiplying by 12 the respondent's income in the month prior to the survey interview. For program group 
members, it includes Family Rewards payments earned during Years 2 and 3. The federal poverty level was calculated based on annual income (monthly 
income multiplied by 12) and the household size at the time of the survey. The poverty threshold was measured according to the 2011 Poverty Guidelines, 
depending on when a respondent was interviewed. 

“Happiness was measured using the U.S. General Social Survey question: "Taken all together, how would you say things are these days — would you say 
that you are very happy, pretty happy, or not too happy?" 

d The Regents exam measures in this table include the following Regents exams: English, Math A, Math B, Geometry, Integrated Algebra, Algebra 
2/Trigonometry, U.S. History and Government, Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

“The End-of-Course exam measures in this table include the following exams: English 1, English, 2, English 3, Biology, Algebra 1, Algebra 2, and U.S. 
History. 

f Attendance was calculated as a percentage of total days present divided by total days enrolled according to district records. Records provided for students 
in New York City include enrollment for the regular school year. Records for students in Memphis include enrollment during the regular school year, 
alternative education programs, and summer school. 

g Child-related health insurance measures were calculated for sample members with at least one child at the time of the survey. 

h The lowest and highest amounts earned in in all years combined were $40 and $19,521 in the Bronx and $40 and $18,400 in Memphis. 



Table 7.2 


Summary Program Impacts, by Poverty Level at Random Assignment 


Income Less Than 50% of FPL at Baseline Income at or Above 50% of FPL at Baseline 

Program Control Difference Program Control Difference 


Outcome 

Group 

Group 

(Impact) 

P-Value 

Group 

Group 

(Impact) 

P-Value 


Income, poverty, and well-being (%) 

Household income at or below the federal poverty 
level (including rewards) a,b 

79.4 

83.4 

-4.1 

0.110 

68.0 

73.3 

-5.3 * 

0.065 


Any savings 

16.6 

10.3 

6 4 *** 

0.004 

24.3 

13.8 

10.5 *** 

0.000 


Any housing/utilities material hardship in 

the past 12 months 

62.5 

64.5 

-2.0 

0.516 

66.2 

63.5 

2.7 

0.371 


Did not have enough money to buy food 

sometime in the past 12 months 

41.9 

48.1 

-6.2 * 

0.052 

44.8 

46.4 

-1.7 

0.590 


Very or pretty happy 0 

75.9 

69.5 

6.3 ** 

0.026 

76.6 

74.7 

1.9 

0.468 


Education 

Number of state core exams passed, Years 1 to 3 d ’° 

2.3 

2.4 

-0.1 

0.329 

3.0 

2.9 

0.1 

0.522 


f 

Attendance rate is 95% or higher, Y ear 3 (%) 

26.2 

27.5 

-1.3 

0.604 

31.4 

30.7 

0.7 

0.787 


Graduated on time (%) 

57.9 

61.5 

-3.5 

0.192 

68.8 

64.6 

4.3 * 

0.098 

ft 

Health 

All dependent children had health insurance 8 (%) 

97.6 

94.7 

2.8 ** 

0.024 

93.1 

94.9 

-1.9 

0.213 

ft 

Parents' average self-rated health 

(1 = poor; 5 = excellent) 

3.0 

2.9 

0.1 * 

0.093 

3.3 

3.1 

0.2 ** 

0.010 


Focal child had 2 or more dental checkups in past 

12 months (%) 

60.2 

48.0 

12.2 *** 

0.000 

65.2 

44.2 

21.0 *** 

0.000 

f 
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Table 7.2 (continued) 


Income Less Than 50% of FPL at Baseline Income at or Above 50% of FPL at Baseline 

Program Control Difference Program Control Difference 


Outcome 

Group 

Group 

(Impact) 

P- Value 

Group 

Group 

(Impact) 

P-Value 

Work 

Ever participated in any education, training, 
or employment activity (%) 

33.5 

31.9 

1.6 

0.590 

39.1 

32.2 

6.9 ** 

0.021 

Currently employed at the time of the survey (%) 

36.9 

37.6 

-0.7 

0.799 

74.1 

75.8 

-1.7 

0.497 

Unemployment insurance-covered employment 
earnings. Years 1 to 3 ($) 

8,239 

11,323 

-3,083 *** 

0.003 

45,955 

47,185 

-1,230 

0.360 

Reward participation. Years 1 to 4 ($) 

Among families who earned at least one reward 
Average reward amount earned h 

5,328 




7,103 




Average reward amount earned, by domain 

Education 

2,767 




3,111 




Health 

1,907 

— 

— 

— 

1,713 

— 

— 

— 

Work 

2,515 

— 

— 

— 

3,101 

— 

— 

— 

Sample size (total = 2,016) 

491 

490 



528 

501 
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Table 7.2 (continued) 


SOURCES: MDRC calculations using data from the Family Rewards 24-month survey, Children's Aid Society's Family Rewards program data. New York 
City Department of Education and Shelby County Schools administrative records, New York State unemployment insurance (Ul) wage records, and 
Tennessee Department of Labor and Workforce Development UI wage records. 

NOTES: FPL = federal poverty level. 

This table presents survey outcomes only for focal children who were living in the household at the time of the interview and at random assignment. 

Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random assignment characteristics of sample members. Standard 
errors were adjusted to account for multiple observations per family. The p-value indicates the likelihood that the difference between the program and 
control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance levels across subgroups are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Education measures include all students who were enrolled in ninth or tenth grade at baseline (2,676 sample members). Employment measures 
calculated from unemployment administrative records include all enrolled parents (2,565 sample members). 

a Monthly household income amounts equal to or greater than $10,000 were excluded from this calculation. About 4.9 percent of the sample was 
excluded from the income measures because respondents did not know or refused to provide the information. An additional 0.2 percent of the sample was 
excluded because the income provided was over $10,000. 

b Annual household income was calculated by multiplying by 12 the respondent's income in the month prior to the survey interview. For program group 
members, it includes Family Rewards payments earned during Years 2 and 3. The federal poverty level was calculated based on annual income (monthly 
income multiplied by 12) and the household size at the time of the survey. The poverty threshold was measured according to the 2011 Poverty Guidelines, 
depending on when a respondent was interviewed. 

c Happiness was measured using the U.S. General Social Survey question: "Taken all together, how would you say things are these days — would you 
say that you are very happy, pretty happy, or not too happy?" 

d The Regents exam measures in this table include the following Regents exams: English, Math A, Math B, Geometry, Integrated Algebra, Algebra 
2/Trigonometry, U.S. Flistory and Government, Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

e The End-of-Course exam measures in this table include the following exams: English 1, English, 2, English 3, Biology, Algebra 1, Algebra 2, and U.S. 
History. 

f Attendance was calculated as a percentage of total days present divided by total days enrolled according to district records. Records provided for 
students in New York City include enrollment for the regular school year. Records for students in Memphis include enrollment during the regular school 
year, alternative education programs, and summer school. 

g Child-related health insurance measures were calculated for sample members with at least one child at the time of the survey. 

h The lowest and highest amounts earned in in all years combined were $40 and $20,700 for families with income less than 50 percent of the federal 



Impacts by Parents’ Education Level 

Table 7.3 presents effects by parents’ educational level, or by whether at least one parent had a 
high school diploma or equivalency certificate or higher at study entry. Families with more 
educated parents earned more than those with less educated parents ($6,515 versus $5,819), 
with the difference due to higher reward payments in the education and work domains. 

The rates for the control groups show that educational level is a strong predictor of 
many of the outcomes shown. More educated parents had much lower poverty rates, their 
children did better in school, they were more likely to work, and they reported somewhat better 
health. Overall, Family Rewards 2.0 had very similar effects across the two types of families. 
The program led to a larger reduction in Ul-covered earnings for the more educated group, but 
the difference between the two impacts is not statistically significant. 


Impacts by Parents’ Employment Status 

Table 7.4 presents effects by parents’ employment status, or whether they were working at 
study entry. The research team defined two-parent families as employed if either parent was 
working at study entry. The table shows that employment status is an important predictor of 
how much families earn from the program. Families with employed parents earned over $7,500 
on average from the program, compared with only $4,800 for other families. This difference is 
due largely to earnings from work rewards. However, families with employed parents also 
earned more in education rewards than those with unemployed parents ($3,180 and $2,685, 
respectively). 

Consistent with the higher reward earnings, the program led to a large reduction in pov- 
erty for the employed group (reducing the rate by 9 percentage points) and had little effect on 
this outcome for the unemployed group. (Although not shown, the program similarly had no 
effect on average household income for the unemployed group.) Another difference in effects is 
that the program led to larger effects on children’s health coverage for the unemployed group. 
This pattern matches the positive effects for the low-income group shown in Table 7.2. In the 
education domain, the program led to a small, positive effect on graduation rates for the em- 
ployed group, similar to the positive effects on this outcome for the higher-income group shown 
in Table 7.2. These two impacts were related, given the substantial overlap between the higher- 
income group and the employed group at study entry. 1 Finally, effects on work were roughly 
similar, with a pattern of reductions in Ul-covered earnings for both groups. 


'For 


example, over 75 percent of higher-income group members were also employed at study entry. 
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Table 7.3 


Summary Program Impacts, by Adult Education at Baseline 


No High School Diploma or High School Diploma or 

Equivalency Certificate Equivalency Certificate 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Income, poverty, and well-being (%) 

Household income at or below the federal poverty 
level (including rewards) a,b 

80.9 

86.7 

-5.8 ** 

0.032 

68.5 

73.0 

-4.5 * 

0.086 

Any savings 

15.2 

9.0 

6.2 *** 

0.009 

24.3 

14.1 

10.2 *** 

0.000 

Any housing/utilities material hardship in 
the past 12 months 

59.0 

64.1 

-5.1 

0.151 

67.6 

64.2 

3.5 

0.204 f 

Did not have enough money to buy food 
sometime in the past 12 months 

42.4 

53.7 

-113 *** 

0.002 

44.2 

43.5 

0.7 

0.817 fff 

Very or pretty happy L 

72.8 

67.7 

5.1 

0.123 

78.4 

74.9 

3.5 

0.142 

Education 

Number of state core exams passed, Years 1 to 3 d,e 

2.5 

2.4 

0.1 

0.479 

2.7 

2.8 

-0.1 

0.211 

Attendance rate is 95% or higher, Year 3 f (%) 

24.4 

23.5 

0.8 

0.765 

31.4 

32.4 

-1.1 

0.654 

Graduated on time (%) 

57.8 

59.0 

-1.2 

0.703 

66.8 

65.7 

1.1 

0.640 

Health 

All dependent children had health insurance 8 (%) 

95.8 

95.6 

0.2 

0.881 

95.2 

93.8 

1.4 

0.296 

Parents' average self-rated health 
(1 = poor; 5 = excellent) 

3.0 

2.9 

0.2 ** 

0.041 

3.2 

3.1 

0.1 ** 

0.025 

Focal child had 2 or more dental checkups in past 

12 months (%) 

61.9 

47.0 

14 9 *** 

0.000 

62.9 

45.7 

17 2 *** 

0.000 


(continued) 





Table 7.3 (continued) 


No High School Diploma or High School Diploma or 

Equivalency Certificate Equivalency Certificate 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Work 

Ever participated in any education, training, 
or employment activity (%) 

32.2 

29.3 

2.9 

0.377 

39.1 

33.7 

5.4 ** 

0.047 

Currently employed at the time of the survey (%) 

48.8 

50.5 

-1.8 

0.542 

60.4 

60.7 

-0.3 

0.909 

Unemployment Insurance-covered employment 
earnings, Years 1 to 3 ($) 

20,735 

21,715 

-980 

0.318 

32,287 

34,941 

-2,655 ** 

0.040 

Reward participation. Years 1 to 4 ($) 

Among families who earned at least one reward 
Average reward amount earned h 

5,819 




6,515 




Average reward amount earned, by domain 

Education 

2,789 




3,021 




Health 

1,858 

— 

— 

— 

1,776 

— 

— 

— 

Work 

2,734 

— 

— 

— 

3,033 

— 

— 

— 

Sample size (total = 1,996) 

396 

373 



618 

609 
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Table 7.3 (continued) 


SOURCES: MDRC calculations using data from the Family Rewards 24-month survey, Children's Aid Society's Family Rewards program data. New York 
City Department of Education and Shelby County Schools administrative records, New York State unemployment insurance (Ul) wage records, and 
Tennessee Department of Labor and Workforce Development UI wage records. 

NOTES: This table presents survey outcomes only for focal children who were living in the household at the time of the interview and at random 
assignment. For students with two parents in the household, no high school diploma or equivalency certificate is defined as having no adults in the 
household with a degree. 

Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. Standard 
errors were adjusted to account for multiple observations per family. The p-value indicates the likelihood that the difference between the program and 
control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance levels across subgroups are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Education measures include all students who were enrolled in ninth or tenth grade at baseline (2,676 sample memebers). Employment measures 
calculated from unemployment administrative records include all enrolled parents (2,565 sample members). 

a Monthly household income amounts equal to or greater than $10,000 were excluded from this calculation. About 4.9 percent of the sample was 
excluded from the income measures because respondents did not know or refused to provide the information. An additional 0.2 percent of the sample was 
excluded because the income provided was over $10,000. 

b Annual household income was calculated by multiplying by 12 the respondent's income in the month prior to the survey interview. For program group 
members, it includes Family Rewards payments earned during Years 2 and 3. The federal poverty level was calculated based on annual income (monthly 
income multiplied by 12) and the household size at the time of the survey. The poverty threshold was measured according to the 2011 Poverty Guidelines, 
depending on when a respondent was interviewed. 

c Happiness was measured using the U.S. General Social Survey question: "Taken all together, how would you say things are these days — would you 
say that you are very happy, pretty happy, or not too happy?" 

d The Regents exam measures in this table include the following Regents exams: English, Math A, Math B, Geometry, Integrated Algebra, Algebra 
2/Trigonometry, U.S. History and Government, Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

e The End-of-Course exam measures in this table include the following exams: English 1, English, 2, English 3, Biology, Algebra 1, Algebra 2, and U.S. 
History. 

' Attendance was calculated as a percentage of total days present divided by total days enrolled according to district records. Records provided for 
students in New York City include enrollment for the regular school year. Records for students in Memphis include enrollment during the regular school 
year, alternative education programs, and summer school. 

^Child-related health insurance measures were calculated for sample members with at least one child at the time of the survey. 

h The lowest and highest amounts earned in in all years combined were $40 and $20,700 for families with no adults with a high school diploma or 
equivalency certificate at baseline and $40 and $22,185 for families who had at least one adult with a high school diploma or equivalency certificate at 
baseline. 



Table 7.4 


Summary Program Impacts, by Adult Employment at Baseline 


Not Employed at Baseline Employed at Baseline 

Program Control Difference Program Control Difference 


Outcome 

Group 

Group 

(Impact) 

P-Value 

Group 

Group 

(Impact) 

P-Value 


Income, poverty, and well-being (%) 

Household income at or below the federal poverty 
level (including rewards) a,b 

82.1 

82.3 

-0.2 

0.932 

65.6 

74.7 

_9 o *** 

0.002 

ft 

Any savings 

16.7 

8.9 

j 9 *** 

0 

24.1 

15.2 

8.9 *** 0.000 


Any housing/utilities material hardship in 
the past 12 months 

62.7 

64.9 

-2.2 

0.48 

65.4 

63.1 

2.3 

0.449 


Did not have enough money to buy food 
sometime in the past 12 months 

41.3 

46.4 

-5.1 

0.105 

45.6 

47.7 

-2.1 

0.503 


Very or pretty happy L 

74.7 

69.6 

5.2 * 

0.072 

77.6 

74.5 

3.2 

0.224 


Education 

d Q 

Number of state core exams passed, Y ears 1 to 3 ’ 

2.2 

2.3 

-0.1 

0.405 

3.1 

3.0 

0.1 

0.348 


f 

Attendance rate is 95% or higher, Y ear 3 (%) 

24.8 

24.8 

0.0 

0.994 

32.6 

33.0 

-0.4 

0.862 


Graduated on time (%) 

58.3 

62.0 

-3.8 

0.172 

68.9 

63.9 

5.0 * 

0.052 

ft 

Health 

All dependent children had health insurance 8 (%) 

96.9 

94.6 

2.3 * 

0.078 

94.2 

95.1 

-1.0 

0.494 

f 

Parents' average self-rated health 
(1 = poor; 5 = excellent) 

3.0 

2.9 

0.1 

0.153 

3.3 

3.1 

q 2 *** 

0.005 


Focal child had 2 or more dental checkups in past 

12 months (%) 

58.4 

43.6 

14.8 *** 

0.000 

66.3 

48.7 

12 ^ *** 

0.000 
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Table 7.4 (continued) 




Not Employed at Baseline 



Employed at Baseline 



Program 

Control 

Difference 


Program 

Control 

Difference 


Outcome 

Group 

Group 

(Impact) 

P- Value 

Group 

Group 

(Impact) 

P- Value 

Work 

Ever participated in any education, training, 
or employment activity (%) 

32.8 

32.3 

0.5 

0.875 

39.3 

32.4 

6.9 ** 

0.021 

Currently employed at the time of the survey (%) 

30.0 

29.1 

0.9 

0.758 

80.4 

83.2 

-2.8 

0.249 

Unemployment Insurance-covered employment 
earnings, Years 1 to 3 ($) 

10,055 

13,304 

-3,249 *** 

0.003 

44,918 

45,477 

-559 

0.667 

Reward participation. Years 1 to 4 ($) 

Among families who earned at least one reward 









Average reward amount earned h 

Average reward amount earned, by domain 

4,839 

— 

— 

— 

7,517 

— 

— 

— 

Education 

2,685 

— 

— 

— 

3,180 

— 

— 

— 

Health 

1,822 

— 

— 

— 

1,795 

— 

— 

— 

Work 

1,888 

— 

— 

— 

3,282 

— 

— 

— 

Sample size (total = 2,006) 

479 

493 



541 

493 




(continued) 
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Table 7.4 (continued) 


SOURCES: MDRC calculations using data from the Family Rewards 24-month survey, Children's Aid Society's Family Rewards program data, New York 
City Department of Education and Shelby County Schools administrative records. New York State unemployment insurance (UI) wage records, and 
Tennessee Department of Labor and Workforce Development UI wage records. 

NOTES: This table presents survey outcomes only for focal children who were living in the household at the time of the interview and at random 
assignment. For students with two parents in the household, not employed at baseline is defined as neither adult being employed at baseline. 

Sample sizes may vary because of missing values. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre -random assignment characteristics of sample members. Standard 
errors were adjusted to account for multiple observations per family. The p-value indicates the likelihood that the difference between the program and 
control groups arose by chance. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance levels across subgroups are indicated as follows: fff = 1 percent; j" f = 5 percent; f = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Education measures include all students who were enrolled in ninth or tenth grade at baseline (2,676 sample memebers). Employment measures 
calculated from unemployment administrative records include all enrolled parents (2,565 sample members). 

“Monthly household income amounts equal to or greater than $10,000 were excluded from this calculation. About 4.9 percent of the sample was 
excluded from the income measures because respondents did not know or refused to provide the information. An additional 0.2 percent of the sample was 
excluded because the income provided was over $10,000. 

b Annual household income was calculated by multiplying by 12 the respondent's income in the month prior to the survey interview. For program group 
members, it includes Family Rewards payments earned during Years 2 and 3. The federal poverty level was calculated based on annual income (monthly 
income multiplied by 12) and the household size at the time of the survey. The poverty threshold was measured according to the 2011 Poverty Guidelines, 
depending on when a respondent was interviewed. 

“Happiness was measured using the U.S. General Social Survey question: "Taken all together, how would you say things are these days — would you 
say that you are very happy, pretty happy, or not too happy?" 

d The Regents exam measures in this table include the following Regents exams: English, Math A, Math B, Geometry, Integrated Algebra, Algebra 
2/Trigonometry, U.S. History and Government, Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

“The End-of-Course exam measures in this table include the following exams: English 1, English, 2, English 3, Biology, Algebra 1, Algebra 2, and U.S. 
History. 

f Attendance was calculated as a percentage of total days present divided by total days enrolled according to district records. Records provided for 
students in New York City include enrollment for the regular school year. Records for students in Memphis include enrollment during the regular school 
year, alternative education programs, and summer school. 

Child-related health insurance measures were calculated for sample members with at least one child at the time of the survey. 

h The lowest and highest amounts earned in in all years combined were $40 and $16,1400 for families with no adults employed at baseline and $40 and 
$22,185 for families who had at least one adult employed at baseline. 



Conclusion 

The results indicate that overall the program had similar effects across the two cities, reducing 
poverty, improving health status and increasing dental visits, and having little effect on school 
progress. The program led to slight reductions in work in both cities, although only in survey- 
reported work in the Bronx and only in Ul-covered work in Memphis. 

The results also suggest that the program had broadly consistent effects across more 
versus less disadvantaged families, with a few differences. For most groups, the program 
increased incomes and reduced poverty. And for all groups, the program increased health status 
and health care use, primarily dental visits. Although the effects on health were fairly similar, 
findings presented in Chapter 5 suggest that the program may have had larger positive effects 
on health for parents who entered the study in poorer health. 

The most notable difference found in the subgroup analysis was the one between em- 
ployed and unemployed parents at study entry. Families without a working parent earned 
substantially less from the program than other families, from both work and education rewards. 
If a main goal of the program is poverty reduction, these parents in particular may need addi- 
tional assistance to earn rewards. Finally, there is some suggestion that the program increased 
graduation rates for a subset of students from more advantaged families. 
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Chapter 8 

Conclusion 


In the ongoing effort to determine what works to help improve the lives of low-income families 
in the United States, conditional cash transfer (CCT) programs represented an exciting and 
innovative approach. The model has had success in lower- and middle-income countries. But 
until 2007, when Family Rewards 1.0 was launched, no comprehensive CCT program had been 
attempted in a higher-income country. Modeled on the programs in other countries, and tested 
in six of New York City’s highest-poverty neighborhoods, Family Rewards 1.0 provided cash 
assistance to low-income families if they met certain conditions related to family health care, 
children’s education, and parents’ work. While the program left the most important outcomes 
unchanged, it did lead to notable reductions in poverty in the short term and increased gradua- 
tion rates for a group of ninth-graders who were more academically prepared at study entry. 

The early findings from the Family Rewards 1.0 evaluation — about how families ex- 
perienced the program, which types of families earned rewards, and which types of families and 
students benefited — suggested ways in which the model could be modified to be more effec- 
tive. Family Rewards 2.0 was the next iteration of the model. The new model offered fewer 
rewards in each of the three areas of children’s education, family health, and parents’ work (8 
rewards total, instead of 22), in order to make the program easier to understand and to focus 
families’ attention on a limited number of outcomes. It offered the education rewards only to 
high school students, given the lack of effects found for younger students, and it made the 
rewards more timely, and thus more salient to families, by paying them monthly for rewards 
earned, rather than every two months, and by rewarding students for grades earned. The biggest 
change, however, was that the new model offered guidance, in which staff members actively 
engaged families in conversations about strategies to earn rewards. 

In short, the model moved from one with a large set of incentives but little guidance to 
one with a smaller set of rewards, that were paid on a timelier basis, and that combined the 
rewards with family guidance. To assess whether this new approach was more effective, the 
program was implemented and tested in the Bronx, New York, and Memphis, Tennessee in July 
2011. Among the group of low-income families who were recruited for and volunteered to 
participate in the study, a randomly chosen subset was offered the new program for three years. 
This report presents the effects on families for three to four years after study entry. 
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Summary of Findings 

After some recruitment and start-up challenges in Year 1, the program was operating generally 
as envisioned in both cities by the middle of Year 2. The family guidance component changed 
the most over time, starting out as fairly modest and transactional during Year 1 and becoming 
more intensive during Year 2, with the introduction of motivational interviewing. In addition, 
the advisors began targeting low-eaming participants to provide them with more guidance 
sessions in the second half of Year 2. Reaching out to these participants involved a tremendous 
amount of effort from advisors. 1 

Further changes during Year 3 included a greater push to have advisors make referrals. 
It was anticipated at the outset that advisors would function as “resource experts” who devel- 
oped a knowledge of the services available in their communities and connected participants to 
appropriate agencies, when such services might help participants meet the conditions to earn 
rewards. However, through Year 2, the fraction of guidance sessions that included referrals 
remained quite low. After the increased effort, referrals increased dramatically in the Bronx in 
Year 3, but remained low in Memphis. 

Overall, participants reported good relationships with their advisors, suggesting that the 
quality of advisement improved over the three years. However, the frequency of advisement 
was not as high as program designers had hoped, particularly in Memphis. Across both cities, 
for example, most participants did not attend the anticipated two fonnal guidance sessions in 
Year 2 or 3, with rates of just over 60 percent in the Bronx and less than 25 percent in Memphis. 
Finally, interviews suggest that most families understood the program fairly well. Reward 
earnings rates by different types of families also suggest that the outreach later in the program 
targeting low reward-earners did lead to increased reward receipt among more disadvantaged 
families, something with which administrators in the first program struggled. Families earned 
on average more than $2,000 per year in rewards in Years 2 and 3. 

In tenns of effects, the findings show that the new program achieved many of the same 
effects as the original model, but fell short in other, important ways. Family Rewards 2.0 met its 
short-term goals of increasing income and reducing poverty for all families and across a range 
of family types. It also increased dental visits and adults’ self-reported health status, particularly 
for those in poorer health at study entry. Similar to Family Rewards 1.0, the new model led to 
reductions in work and earnings for some participants. However, in contrast to the original 
program, it did not affect students’ school progress through Year 4, either for the full sample of 
students or for the subgroup of academically proficient students. Although there were some 


See Dechausay et al. (2014) for more details about program implementation during the first two years. 
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small, positive effects on attendance and credits for the ninth-graders, these effects did not 
persist or lead to lasting effects on enrollment or graduation. 

The addition of Memphis as a site in the study also provided information on how the 
program might work in a different context, since the original program had been tested only in 
New York City. In general, the program was successfully implemented in both cities and had 
similar effects on participating families’ poverty, health, education, and work. However, the 
context did matter for how the program was implemented and how families experienced it. In 
some cases, the differences were due to specific policies. For example, students in the Bronx 
were less likely to earn rewards for attendance during the first two years because New York 
City public schools count tardiness toward absences, and program staff followed that conven- 
tion until Year 3. In contrast, more students in the Bronx earned the higli- value reward for 
passing the state core exams because they were well informed about the exams and took 
practice tests in class as preparation. In Memphis, these exams were relatively new. As another 
example, fewer adults claimed the reward for dental checkups in Memphis than in the Bronx, 
largely because the public health insurance program in Tennessee does not cover dental clean- 
ings, as it does in New York State. 

While the program might have been easily adjusted for these types of local differences, 
what proved to be more challenging in Memphis was transportation. In that city, the lack of 
good public transit was an important barrier to participation in the guidance component. Staff 
members adapted by conducting guidance sessions over the phone or by making home visits, 
but they did not have much direct contact with students. Participants in Memphis tended to feel 
that they were not as close to their advisors as did participants in the Bronx. While the findings 
suggest that these differences in the quality and quantity of advisement did not lead to different 
effects, it is nonetheless an important issue to consider in a model in which guidance is a central 
component. 

Finally, an analysis of program expenditures found that the program cost just under 
$13,500 per family over the full program period. Reward payments accounted for about half of 
this amount, and the costs of operating the program, which included administering the payments 
and providing the advisement component, accounted for the other half. As noted in Chapter 2, 
these administrative costs were undoubtedly higher than what such costs would have been had 
the program been operated on a much larger scale and not as part of a relatively small demon- 
stration project. 


Lessons from Family Rewards 

Although a somewhat modified replication of the original model that targeted different types of 
families, Family Rewards 2.0 adds to the evidence about how a CCT program might work in 
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the United States. By comparing the findings from the two Family Rewards evaluations, it is 
possible to draw some general conclusions about this type of holistic CCT program. 

Most families earn rewards, especially in education and health. In terms of reward 
receipt, nearly all enrolled families earned at least some rewards in both programs, and most of 
their earnings came from the education and health domains. For example, over 97 percent of 
families in each program earned at least one education reward, about 98 percent in the first 
program and 90 percent in the second program earned at least one health reward, and just over 
50 percent in each program earned at least one work reward. Family Rewards 2.0 did not 
include rewards for maintaining health insurance, so the relatively high receipt rate in that 
domain is encouraging. 

More disadvantaged families earn less than other fa mili es. One of the goals of the 
targeted outreach strategy in Family Rewards 2.0, in which staff reached out to families whose 
reward earnings were low, was to increase reward receipt among the types of families who had 
earned less in Family Rewards 1.0. Comparing the receipt rates of similar rewards across both 
programs, the data on reward earnings suggest that this outreach succeeded. However, the 
general pattern that more advantaged families earned substantially more in rewards than their 
disadvantaged counterparts still held in both programs. Parents of families that earned more 
rewards, for example, had higher education levels, were more likely to be working, and were 
more likely to be married than those of families that earned less. 

Family Rewards reduces poverty and material hardship and increases savings 
while the rewards are offered. But a program with fewer rewards transfers less cash to 
families and leads to smaller effects. Family Rewards 2.0 included fewer rewards in order to 
make the program easier to understand and focus families’ effort on key milestones. The 
qualitative data suggest that families understood the more streamlined program more easily, 
although the result of offering fewer rewards was that families earned on average much less per 
year (just over $2,000 per year in Family Rewards 2.0, compared with about $3,000 per year in 
Family Rewards 1.0). Not surprisingly, the program reduced poverty by lesser a degree (by 5 
percentage points in Family Rewards 2.0 compared with 12 percentage points in Family 
Rewards 1 .0) and had fewer effects on several measures of material hardship. 

Family Rewards increases dental visits and seems to improve some parents’ health 
status. The two programs had fairly similar effects on health. Both had the largest and most 
consistent effects on dental visits, with both parents and children much more likely to have 
completed the two recommended visits per year. Both programs also seemed to improve 
parents’ self-rated health status. Family Rewards 1.0 had a marginally significant effect on this 
outcome for the full sample, but a statistically significant effect for those in poor health at study 


118 



entry. Family Rewards 2.0 replicated this finding. Neither program led to changes in health 
coverage or preventive medical visits. 

Family Rewards leads to modest reductions in work, particularly among more 
disadvantaged groups. Although other work incentives programs have had positive effects on 
work, there was a concern about Family Rewards from the outset that families’ reward earnings 
in the other two domains might create an “income effect” and lead to reductions in work. An 
income effect was observed to some extent in both programs, although in somewhat different 
ways. The first program led to a modest increase in employment for the sample as a whole, but 
largely in work not captured by the unemployment insurance (UI) system. It also led to notable 
reductions in Ul-covered work and earnings for parents without a high school diploma and 
those with no recent work experience. The second program led to reductions in Ul-covered 
earnings for the full sample, although negative effects in Memphis largely drove this change. 
The reasons for the differences across the two cities are not entirely clear, but the findings add 
to the evidence that CCT programs with generous incentives have the unintended effect of 
reducing work to some extent. This reduction in work conflicts with the intent of CCT programs 
to increase longer-term self-sufficiency. 

Family Rewards can sometimes improve school outcomes. The original program led 
to notable increases in school progress and graduation rates for ninth-graders who entered the 
study already proficient in English language arts. The hope with the modified model, particular- 
ly through its rewards for grades and the addition of family guidance, was that it would lead to 
similar effects, but for all students, including those who were struggling academically. With the 
exception of some small effects for entering ninth-graders, Family Rewards 2.0 did not general- 
ly affect school progress for the full group of students and was not able to replicate the positive 
effect for proficient students. 

Although the students in Family Rewards 2.0 were from different types of families (for 
instance, they were from families receiving Temporary Assistance for Needy Families or 
Supplemental Nutrition Assistance Program benefits and from families living in Memphis in 
addition to the Bronx), these differences did not seem to account for the lack of effects. Another 
way in which students in the two programs differed is that students in the Family Rewards 2.0 
study were doing better in school in the absence of the program, with higher rates of attendance 
and grade-to-grade retention. By the end of Year 2, for example, only 44 percent of the ninth- 
graders in the Family Rewards 1.0 control group had earned enough credits to stay on track for 
an on-time graduation, compared with 60 percent of ninth-graders in Family Rewards 2.0 
control group. (A later paper will present a more formal analysis of the differences between the 
two programs.) Nonetheless, the findings across both programs show that a CCT program of 
this design, with or without guidance, does not have robust and consistent effects on students’ 
school outcomes. 


119 



Adding more intensive guidance to the model is challenging and does not neces- 
sarily lead to bigger effects. Not surprisingly, providing more intensive guidance to families 
about how to earn rewards was the most challenging part of the program. Although advisors 
built relationships with participants during Year 1, they largely relied on a light-touch and 
transactional method of guidance, reporting that they often felt unqualified to address some of 
the personal issues that families brought up. In Year 2, the advisors received training in motiva- 
tional interviewing and were pushed to provide more intensive guidance and extra outreach to 
the lowest-earning families. Overall, they succeeded on both fronts, although it required 
significant time and effort on their part. The more intensive advisement contributed to higher 
program costs. One way to measure program costs is to calculate how much every dollar of 
reward payments transferred to families cost each program. That cost was $0.47 in Family 
Rewards 1 .0 and $1 .07 in Family Rewards 2.0. 

However, a comparison of effects across both programs suggests that the more inten- 
sive guidance and advising did not lead to bigger or more widespread effects. This result was 
found in Memphis, where transportation barriers prevented advisors from meeting with families 
as often as hoped, but also in the Bronx, where advisors met with parents and students more 
frequently. 


Next Steps 

Evidence from the two Family Rewards programs points to what a CCT model can and cannot 
do for families in the United States. A CCT program can transfer significant sums of money to 
low-income families and reduce poverty. And it can increase dental visits and parents’ health 
status. In terms of the other domains, an important question is: are there any modifications to the 
current model that might lead to stronger and more consistent effects on students’ schooling or 
that might reduce the incentives it creates for parents to reduce their work effort? 

In terms of work, the answer is not necessarily to remove the work rewards, since the 
reductions in work might have been even larger in their absence. Should all or some fraction of 
the rewards (in all domains) be conditional upon work? Doing so might minimize the reductions 
in work, but it would also decrease the income transfer going to the most disadvantaged 
families. Or, is the answer to provide more support for families who want to work? The value of 
more guidance seems to be less clear, given that Family Rewards 2.0 provided it and that many 
families, especially in Memphis, had a hard time taking advantage of it. 

More guidance might be one answer to making the education rewards more effective. 
But the issue of participation remains, since many students would probably still have trouble 
attending guidance sessions. If more guidance is the answer, and it is not clear that it is, then one 
option might be to make the program school based. If the program were offered to an entire 
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school, the incentives themselves might be more salient (through group and classroom dynam- 
ics), and students who needed tutoring or other assistance might be more likely to get it (assum- 
ing the school has the resources). However, enrolling an entire school in a holistic CCT pro- 
gram — with rewards for reaching milestones in education, health, and work milestones — 
might mean offering rewards to some families who are not low income. This prospect would 
limit the program’s ability to reduce short-term poverty, which is a primary program goal. 

Finding effective ways to invest in families and break the cycle of poverty is critical, 
and the results presented in this report indicate that perhaps Family Rewards is not part of the 
answer. Nonetheless, the benefits of making even small improvements in health and education 
suggest that the continued search is worth the effort. 
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Appendix A 


Rewards Offered in Family Rewards 1.0 and 2.0 




Appendix Table A.l 

Comparison of Rewards Offered by Family Rewards 1.0 and 2.0 




Family Rewards 2.0 

Reward Type 

Family Rewards 1.0 

Bronx 

Memphis 

Education incentives 




Elementary and middle school students (paid to parent) 




Attends 95% of scheduled school days every month 

$25 per month (discontinued after Year 2) 

Not offered 

Not offered 

Scores at proficiency level or improves 
on annual math and English tests 




Elementary school students 

$300 per math test; $300 per English test 

Not offered 

Not offered 

Middle school students 

$350 per math test; $350 per English test 

Not offered 

Not offered 

Parent reviews low-stakes interim tests 
up to 5 times per year 

$25 for parents to download, print, and 
review results (discontinued after Year 1) 

Not offered 

Not offered 

Parent discusses up to 2 annual math and 

English test results with teachers 

$25 (discontinued after Y ear 2) 

Not offered 

Not offered 

High school students 3 




Takes up to 2 PSAT tests 

$50 per test 

Not offered 

Not offered 

Accumulates 1 1 course credits per year 

$600 

Not offered 

Not offered 

Graduates from high school 

$400 

Not offered 

Not offered 

Attends 95% of scheduled school days 

$50 per month 

$40 per month 

$40 per month 

Takes an SAT or ACT exam (once during program) 

Not offered 

$50 

$50; must score 19 or 
better on the ACT if 
administered by MCS b 


(continued) 



Appendix Table A.l (continued) 




Family Rewards 2.0 

Reward Type 

Family Rewards 1.0 

Bronx 

Memphis 

Education incentives (continued) 

Receives grades on an official report card 

Not offered 

$30 per A (90-100); 
$20 per B (80-89); 
$10 per C (75-79) 

$30 per A (93-100); 

$20 per B (85-92); 

$10 per C (75-84) c 

Passes up to 5 Regents exams d or 7 End-of- 
Course exams e 

$600 per Regents exam 

$500 per Regents 
exam for a score of 
75 or above; $400 
per exam for a 
score of 65-74 

$200 per End-of- 
Course exam for a 
score of proficient or 
advanced (increased 
to $300 in Year 2) 

All grades 




Parent attends parent-teacher conferences 
up to 2 times per year 

$25 per conference 

Not offered 

Not offered 

Child obtains library card 

$50 once during program 
(discontinued after Year 2) 

Not offered 

Not offered 

Health incentives 




Maintaining public or private health insurance 


Not offered 

Not offered 

For each parent covered 

Per month: $20 (public); $50 (private) 



If all children are covered 

Per month: $20 (public); $50 (private) 
(discontinued after Year 2) 



Annual medical checkup 

$200 per family member 

$100 per family 
member 

$100 per family 
member 

A doctor-recommended follow-up visit 

$100 per family member 
(discontinued after Year 2) 

Not offered 

Not offered 


(continued) 



Appendix Table A.l (continued) 




Family Rewards 2.0 

Reward Type 

Family Rewards 1.0 

Bronx 

Memphis 

Early intervention evaluation for child under 30 
months old, if advised by pediatrician 

$200 per child 

Not offered 

Not offered 

Preventive dental care every 6 months 
(once per year for children 1-5 years old) 

$100 per family member 

$100 per family 
member 

$100 per family member 

Adult work efforts 




Sustained full-time employment 

$150 per month 

$150 per month 

$150 per month 

Education and training while employed at least 

10 hours per week 

Amount varied by length of course, 
up to a maximum of $3,000 over 3 years 
(employment requirement discontinued 
after Year 2) 

Not offered 

Not offered 

GED certificate 

Not offered 

$400 for successful 
completion 

$400 for successful 
completion 


SOURCE: Children’s Aid Society's Family Rewards 2.0 program materials and Seedco's Family Rewards 1.0 program materials. 

NOTES: MCS = Memphis City Schools; GED = General Education Development. 

“In Family Rewards 1.0, the following education rewards were paid entirely to students: taking up to two PSAT Tests ($50), passing five Regents exams 
($600 per exam), and obtaining a library card ($50). The following education rewards were split between parents and students: accumulating 1 1 course credits 
per year ($600) and attending 95 percent of scheduled school days. In Family Rewards 2.0, all education rewards were paid entirely to students. 

b The ACT is scored out of 36. Memphis City Schools officials requested that the minimum score for the reward be set at 19 for students taking 
the test for free in class because this score is considered an indication that students are ready for college-level work. 

c ln Year 1, the amount of each reward for grades was prorated based on the number of official report cards issued by a student's school. To 
simplify verification in Year 2, students were paid the listed amounts for their grades regardless of the number of official report cards they 
received, up to a maximum of $600 per program year. 

d High school students (grades 9-12) in the Bronx were eligible to earn rewards for the following Regents exams: English, one of any math exams 
(including Math A, Math B, Integrated Algebra, Geometry, and Algebra 2/Trigonometry), U.S. Flistory and Government, Global Flistory and Geography, 
and one of any science exams (including Living Environment, Chemistry, Physics, and Earth Science). 

TTigh school students (grades 9-12) in Memphis were eligible to earn rewards for the following End-of-Course exams: Algebra 1, Algebra 2, Biology, 
English 1, English 2, English 3, and U.S. Flistory. 




Appendix B 


Parents’ Understanding of and Experience with 

Family Rewards 




Appendix Table B.l 

Parents' Understanding of and Experience with the Family Rewards Incentives and 

Procedures 

Outcome (%) 

Program 

Group 

Across all domains 3 

Average proportion of rewarded activities that respondents correctly identified 

b c 

as eligible for rewards ’ 

89.4 

Average proportion of unrewarded activities that respondents correctly identified 
as not eligible for rewards' 1 

35.3 

Payment process 

Agreed or strongly agreed with the following statements 

My payments from the program are always on time 

1 have given up trying to earn certain rewards because it is hard to get paid for things 1 do 

1 remember the deadlines for the coupons and get them filled out completely 

When 1 don't earn a reward for an activity, 1 understand why 

81.2 

41.6 

87.9 

91.1 

Sample size 

1,025 


SOURCE: MDRC calculations using data from the Family Rewards 24-month survey. 

NOTES: This table excludes control group members because it pertains only to the Family Rewards 
program. 

Sample sizes may vary because of missing values. 

Responses of "don't know" are considered a wrong answer. 

b This measure refers to the average number of correctly identified items divided by the total number 
of items and multiplied by 100. 

c There are eight rewarded activities that respondents could identify as eligible. 
d There are three unrewarded activities that respondents could identify as not eligible. 
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Appendix C 


Impacts on Public Benefit Receipt 




Appendix Table C.l 

Impacts on Temporary Assistance for Needy Families (TANF) or Safety Net 
Assistance (SNA) and Food Stamp Receipt and Payments, Years 1 to 2 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Ever received TANF/SNA (%) 

Y ears 1 to 2 

22.8 

23.0 

-0.2 

0.880 

Year 1 

18.1 

17.6 

0.5 

0.729 

Year 2 

14.6 

13.9 

0.7 

0.601 

Amount of TANF/SNA received (S) 

Years 1 to 2 

897 

834 

64 

0.562 

Year 1 

449 

414 

35 

0.525 

Year 2 

448 

420 

28 

0.658 

Ever received food stamps (%) 

Years 1 to 2 

95.8 

95.9 

-0.2 

0.835 

Year 1 

91.2 

91.5 

-0.3 

0.777 

Year 2 

90.1 

89.1 

1.0 

0.389 

Amount of food stamps received ($) 

Y ears 1 to 2 

9,993 

9,589 

404 * 

0.068 

Year 1 

5,292 

5,016 

276 ** 

0.018 

Year 2 

4,700 

4,573 

128 

0.273 

Sample size (total = 2,456) 

1,230 

1,226 




SOURCE: MDRC calculations using administrative records data from the New York City and 
Memphis human resources administrations. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre- 
random assignment characteristics of sample members. Standard errors were adjusted to account 
for multiple observations per family. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Dollar averages include zero values for sample members who were not receiving TANF or SNA 
benefits or food stamps. 
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Appendix Table C.2 


Impacts on Temporary Assistance for Needy Families (TANF) or Safety Net 
Assistance (SNA) and Food Stamp Receipt and Payments, Years 1 to 2 by City 




Bronx 



Memphis 


Program 

Control Difference 

P-Value 

Program 

Control Difference 

P- Value 

Ever received TANF/SNA 1%) 









Y ears 1 to 2 

22.1 

21.3 

0.9 

0.684 

23.6 

24.7 

-1.1 

0.600 

Year 1 

14.6 

13.2 

1.3 

0.440 

21.6 

21.9 

-0.3 

0.876 

Year 2 

17.0 

17.2 

-0.2 

0.926 

12.2 

10.5 

1.7 

0.313 

Amount of TANF/SNA received (SI 









Y ears 1 to 2 

1,290 

1,205 

86 

0.675 

506 

459 

47 

0.437 

Year 1 

559 

521 

39 

0.698 

338 

307 

31 

0.429 

Year 2 

731 

684 

47 

0.701 

168 

152 

16 

0.597 

Ever received food stamps (%) 









Y ears 1 to 2 

98.3 

98.3 

0.0 

1.000 

93.3 

93.5 

-0.2 

0.879 

Year 1 

90.3 

90.2 

0.0 

0.995 

92.1 

92.7 

-0.6 

0.700 

Year 2 

90.9 

92.4 

-1.5 

0.329 

89.3 

85.7 

3.6 * 

0.053 ft 

Amount of food stamps received (S') 









Y ears 1 to 2 

9,479 

9,290 

190 

0.498 

10,489 

9,908 

581 * 

0.085 

Year 1 

5,015 

4,845 

169 

0.269 

5,561 

5,198 

363 ** 

0.036 

Year 2 

4,465 

4,444 

21 

0.890 

4,929 

4,710 

219 

0.220 

Sample size (total = 2,456) 

617 

613 



613 

613 




SOURCE: MDRC calculations using administrative records data from the New York City and Memphis human resources administrations. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. 
Standard errors were adjusted to account for multiple observations per family. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 
5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 
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Impacts on Education Outcomes, by Math Proficiency 




Appendix Table D.l 


Impacts on Enrollment, Graduation, Attendance, Credits, and Regents or End-of-Course Exams 
for Students in Grades 9 and 10 at the Time of Random Assignment, by Math Proficiency Test Score 


Proficient on Eighth-Grade Math Test Not Proficient on Eighth-Grade Math Test 


Grade Level and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 


Enrollment, Year 3 (%) 

On grade 

91.1 

85.5 

5.5 ** 

0.035 

79.5 

79.6 

-0.1 

0.952 

f 

Not on grade 

6.1 

9.6 

-3.5 

0.115 

12.8 

11.9 

0.9 

0.530 

f 

Not enrolled 

2.8 

4.9 

-2.1 

0.201 

7.7 

8.5 

-0.8 

0.518 


Graduation (%) 

Graduated on time 

77.9 

74.6 

3.3 

0.350 

61.9 

62.2 

-0.3 

0.881 


Graduated by Y ear 4 d 

78.6 

76.2 

2.4 

0.491 

64.6 

65.7 

-1.2 

0.599 


Attendance b (%) 

Attendance rate, Y ear 1 

89.3 

86.6 

2.7 

0.204 

87.4 

88.9 

-1.5 ** 

0.042 

t 

Attendance rate, Y ear 2 

87.6 

83.6 

4.0 * 

0.070 

82.7 

82.1 

0.7 

0.546 


Attendance rate, Y ear 3 

84.3 

80.1 

4.1 * 

0.078 

78.4 

77.7 

0.7 

0.595 


Attendance rate is 95% or higher, Y ear 1 

60.1 

52.0 

8.1 * 

0.050 

41.0 

42.8 

-1.9 

0.428 

ft 

Attendance rate is 95% or higher, Y ear 2 

48.7 

47.9 

0.8 

0.837 

34.8 

33.8 

1.0 

0.650 


Attendance rate is 95% or higher, Y ear 3 

36.8 

40.1 

-3.3 

0.404 

28.0 

27.0 

1.1 

0.612 


Credits^ 

Number of credits earned, Year 1 

12.3 

11.7 

0.5 

0.203 

10.7 

11.1 

-0.3 

0.123 

t 

Number of credits earned, Year 2 

11.4 

11.3 

0.1 

0.858 

9.9 

9.8 

0.1 

0.837 


Number of credits earned, Year 3 

10.7 

10.1 

0.6 

0.202 

9.2 

9.1 

0.1 

0.679 


Number of credits earned, Years 1 to 3 

34.3 

33.1 

1.2 

0.308 

29.8 

30.0 

-0.2 

0.785 


Earned adequate credits, Years 1 to 3 (%) 

73.5 

68.9 

4.6 

0.224 

52.5 

53.4 

-0.9 

0.715 
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Appendix Table D.l (continued) 


Proficient on Eighth-Grade Math Test Not Proficient on Eighth-Grade Math Test 


Grade Level and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

d e 

Resents or End-of-Course exams ’ 

State core exams taken, Y ears 1 to 3 

7.5 

7.1 

0.4 

0.108 

6.3 

6.3 

-0.1 

0.760 

State core exams passed, Years 1 to 3 

5.0 

4.9 

0.1 

0.452 

2.0 

2.1 

0.0 

0.728 

Sample size (total = 2,450) 

284 

308 



955 

903 




SOURCES: MDRC calculations using data from New York City Department of Education and Shelby County Schools administrative records. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. 
Standard errors were adjusted to account for multiple observations per family. Statistical significance levels are indicated as follows: *** = 1 
percent; ** = 5 percent; * = 10 percent. Statistical significance levels across subgroups are indicated as follows: ff f = 1 percent; f f = 5 percent; 
f = 10 percent. 

Rounding may cause slight discrepancies in calculating sums and differences. 

Note that all outcomes in the table include zero values for students who were no longer enrolled. 

Years 1, 2, 3, and 4 cover the 2011-2012, 2012-2013, 2013-2014, and 2014-2015 school years, respectively. 

a Students enrolled in tenth grade at the time of random assignment had five years to complete graduation in this measure. Students enrolled in 
ninth grade at the time of random assignment had four years. 

b Attendance was calculated as a percentage of total days present divided by total days enrolled according to district records. Records provided for 
students in New York City include enrollment for the regular school year. Records for students in Memphis include enrollment during the regular 
school year, alternative education programs, and summer school. 

c Students in New York City earn 1 credit per course per semester completed. Students in Memphis earn 0.5 credits per course per semester. 
Credits for students in Memphis were multiplied by two to create a standard scale for comparison. To be considered on time to graduate, students in 
New York City must earn an average of 1 1 credits per school year and students in Memphis must earn an average of 5.5 credits per school year. 

d The Regents exam measures in this table include the following Regents exams: English, Math A, Math B, Geometry, Integrated Algebra, 
Algebra 2/Trigonometry, U.S. History and Government, Global History and Geography, Living Environment, Chemistry, Physics, and Earth 
Science. 

e The End-of-Course exam measures in this table include the following exams: English 1, English, 2, English 3, Biology, Algebra 1, Algebra 2, 
and U.S. History. 



Appendix Table D.2 


Impacts on Enrollment, Graduation, Attendance, Credits, and Regents or End-of-Course Exams 
for Students in Grade 9 at the Time of Random Assignment, by Math Proficiency Test Score 

Proficient on Eighth-Grade Math Test Not Proficient on Eighth-Grade Math Test 

Program Control Difference Program Control Difference 

Grade Level and Outcome Group Group (Impact) P- Value Group Group (Impact) P- Value 


Enrollment and Graduation, Year 4 (%) 

On grade 
Not on grade 
Not enrolled 

Graduated on time 

Attendance 3 (%) 

Attendance rate, Y ear 1 
Attendance rate, Year 2 
Attendance rate, Year 3 
Attendance rate, Y ear 4 

Attendance rate is 95% or higher, Y ear 1 
Attendance rate is 95% or higher, Y ear 2 
Attendance rate is 95% or higher, Y ear 3 
Attendance rate is 95% or higher, Y ear 4 

Credits 1 ' 

Number of credits earned, Year 1 
Number of credits earned, Year 2 
Number of credits earned, Year 3 
Number of credits earned, Year 4 
Number of credits earned, Years 1 to 4 

Earned adequate credits, Years 1 to 4 (%) 


84.2 

76.9 

7.3 * 

0.082 

5.1 

11.2 

-6.2 ** 

0.028 

10.8 

11.9 

-1.1 

0.749 

74.1 

72.8 

1.3 

0.786 


90.0 

87.8 

2.2 

0.367 

86.7 

83.6 

3.1 

0.248 

83.9 

80.1 

3.8 

0.196 

72.5 

65.7 

6.7 * 

0.097 

60.9 

52.6 

8.2 

0.116 

46.7 

47.5 

-0.8 

0.883 

37.0 

43.0 

-6.0 

0.228 

27.5 

21.1 

6.4 

0.154 


12.5 

11.7 

0.8 

0.136 

11.3 

11.2 

0.1 

0.862 

10.9 

10.3 

0.5 

0.360 

9.4 

9.0 

0.3 

0.559 

44.0 

42.3 

1.8 

0.348 

70.8 

67.4 

3.4 

0.479 


68.7 

69.1 

-0.4 

0.896 

12.5 

10.4 

2.1 

0.274 

18.8 

20.5 

-1.7 

0.490 

59.4 

58.8 

0.5 

0.863 


86.5 

87.6 

-1.2 

0.284 

82.5 

80.3 

2.2 

0.142 

79.4 

76.8 

2.6 

0.139 

59.7 

57.6 

2.1 

0.423 

41.1 

41.1 

0.0 

0.989 

35.0 

31.7 

3.3 

0.270 

32.8 

29.9 

3.0 

0.294 

15.0 

16.8 

-1.8 

0.454 


10.3 

10.5 

-0.2 

0.459 

10.0 

9.4 

0.6 * 

0.076 

9.2 

OO 

OC 

0.5 

0.194 

8.7 

8.4 

0.3 

0.437 

38.3 

37.1 

1.2 

0.286 

50.7 

50.4 

0.4 

0.912 


(continued) 


Appendix Table D.2 (continued) 


Proficient on Eighth-Grade Math Test Not Proficient on Eighth-Grade Math Test 


Grade Level and Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P- Value 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

P-Value 

c d 

Resents or End-of-Course exams ’ 

State core exams taken, Y ears 1 to 4 

8.8 

8.0 

0.8 ** 

0.044 

7.4 

7.2 

0.2 

0.594 

State core exams passed, Y ears 1 to 4 

5.8 

5.5 

0.3 

0.341 

2.3 

2.3 

0.0 

0.961 

Sample size (total = 1,400) 

171 

190 



534 

505 




SOURCES: MDRC calculations using data from New York City Department of Education and Shelby County Schools administrative records. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. 
Standard errors were adjusted to account for multiple observations per family. Statistical significance levels are indicated as follows: *** = 1 percent; 

** = 5 percent; * = 10 percent. Statistical significance levels across subgroups are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 
Rounding may cause slight discrepancies in calculating sums and differences. 

Note that all outcomes in the table include zero values for students who were no longer enrolled. 

Years 1, 2, 3, and 4 cover the 2011-2012, 2012-2013, 2013-2014, and 2014-2015 school years, respectively. 

“Attendance was calculated as a percentage of total days present divided by total days enrolled according to district records. Records provided for 
students in New York City include enrollment for the regular school year. Records for students in Memphis include enrollment during the regular school 
year, alternative education programs, and summer school. 

b Students in New York City earn 1 credit per course per semester completed. Students in Memphis earn 0.5 credits per course per semester. Credits 
for students in Memphis were multiplied by two to create a standard scale for comparison. To be considered on time to graduate, students in New York 
City must earn an average of 1 1 credits per school year and students in Memphis must earn an average of 5.5 credits per school year. 

c The Regents exam measures in this table include the following Regents exams: English, Math A, Math B, Geometry, Integrated Algebra, Algebra 
2/Trigonometry, U.S. History and Government, Global History and Geography, Living Environment, Chemistry, Physics, and Earth Science. 

d The End-of-Course exam measures in this table include the following exams: English 1, English, 2, English 3, Biology, Algebra 1, Algebra 2, and 
U.S. History. The U.S. History exam was not administered for students in Year 4. 
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